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Abstract 

The Collider Detector at Fermilab (CDF) pursues a broad physics program 
at Fermilab's Tevatron collider. Between Run II commissioning in early 2001 
and the end of operations in September 2011, the Tevatron delivered 12 fb _1 
of integrated luminosity of pp collisions at ^fs = 1.96 TeV. The physics at 
CDF includes precise measurements of the masses of the top quark and W 
boson, measurement of CP violation and B s mixing, and searches for Higgs 
bosons and new physics signatures, all of which require heavy flavor tagging 
with large charged particle tracking acceptance. To realize these goals, in 2001 
CDF installed eight layers of silicon microstrip detectors around its interaction 
region. These detectors were designed for 2-5 years of operation, radiation 
doses up to 2 Mrad (0.02 Gy), and were expected to be replaced in 2004. The 
sensors were not replaced, and the Tevatron run was extended for several years 
beyond its design, exposing the sensors and electronics to much higher radiation 
doses than anticipated. In this paper we describe the operational challenges 
encountered over the past 10 years of running the CDF silicon detectors, the 
preventive measures undertaken, and the improvements made along the way 
to ensure their optimal performance for collecting high quality physics data. 
In addition, we describe the quantities and methods used to monitor radiation 
damage in the sensors for optimal performance and summarize the detector 
performance quantities important to CDF's physics program, including vertex 
resolution, heavy flavor tagging, and silicon vertex trigger performance. 

Keywords: Silicon, Vertex Detector, CDF, Tevatron Run II, Detector 
Operations 
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1. Introduction 



The Tevatron collider at the Fermi National Accelerator Laboratory (FNAL) 
collided proton and antiproton beams at a center-of-mass energy of 1.96 TeV. 
The collisions happened at two interaction points where multipurpose detectors 
Collider Detector at Fermilab (CDF II) and DO, were positioned. 

The CDF II detector 1] was a general purpose detector with a cylindri- 
cal geometry. The innermost part of the detector consisted of charged-particle 
tracking detectors, shown in Fig.[TJ which were located inside a superconducting 
solenoidal magnet which provided a highly uniform 1.4 T magnetic field oriented 
parallel to the beam axis. Calorimeters and muon systems outside the solenoid 
provided lepton identification and momentum measurement as well as jet en- 
ergy measurements. The tracking detectors and calorimeters together provided 
identification of jets from heavy (charm and bottom) quarks. 

The inner component of the tracking system was a series of silicon microstrip 
detectors that constituted the CDF silicon detector. Beyond the silicon detector 
lay the Central Outer Tracker (COT), an open-cell drift chamber. Together 
with the additional constraints coming from the position of the primary vertex, 
the COT and Silicon Detector provided a resolution on the track momentum 
transverse to the beam direction, px, of (j{pt)/pt = 0.15% • px/(GeV/c). 



CDF Tracking Volume 




Figure 1: Isometric view of the entire CDF II detector (left), and schematic layout of the 
CDF II tracking system (right). 

CDF II used a cylindrical coordinate system with the z axis oriented along 
the proton beam direction and azimuthal angle <fr measured around the beam 
axis. The polar angle 9 was measured with respect to the positive z (proton- 
beam) direction and was used to define the pseudorapidity r\ = — ln(tan(0/2)). 

The physics program of Run II at the Tevatron includes precision measure- 
ments of the mass of the top quark and W boson; bottom and charm physics, 
including the determination of the B s and D° mixing parameters; studies of the 
strong interaction (jet multiplicities, diffractive physics, etc.); and searches for 
objects and phenomena as varied as the Higgs boson, supersymmetric particles, 
hidden space-time dimensions, and quark substructure [2j-|5j. All these measure- 
ments benefit from a high-resolution tracking detector and many rely heavily on 



31 the efficient identification of heavy quarks by detection of displaced secondary 

32 vertices, and are enhanced by the capability to trigger on tracks originating 

33 away from the beam. 

34 The CDF silicon detector was designed to withstand radiation doses up to 

35 2MRad (0.02 Gy), the dose expected during the first 2-5 years of CDF opera- 

36 tions, with an upgrade planned in 2004 @. However, the upgrade project was 

37 canceled in 2003, and Run II was later extended into late 2011, with total deliv- 

38 ered integrated luminosity of 12 fb _1 . Several preventive measures were taken to 

39 keep the original silicon detector operational and maintain its performance. The 

40 most important of these was the decrease in the operating temperature of the 

41 detector, which reduced the impact of chronic radiation exposure (Section [5]) . 

42 Steps were also taken to minimize thermal cycles, damage from resonances of 

43 wire bonds fSection !3.6l) . and instabilities and sudden loss of the Tevatron beams 

44 (Section©. 

45 Issues arising from radiation damage of the sensors, aging infrastructure, 

46 and electronics were addressed continuously in addition to the basic challenges 

47 posed by the inaccessibility of the detector volume and large number of readout 

48 channels (approximately 722,000). The operational challenges, improvements 

49 to, and the performance of the CDF silicon detector are presented in this paper. 

50 This paper is organized as follows: Section [2] provides a general description 

51 of the detector, Section [3] gives an overview of the data acquisition, the trigger, 

52 and the interface between them, Section 2] describes the power supplies and the 

53 operational experience with them and response to their failures, Section [5] dc- 

54 tails the design, history, and response to failures in the cooling system, Section[6] 

55 gives a review of particle beam incidents, and response to them. Section [7] dc- 

56 tails the readout calibration, Section [8] is dedicated to the routine monitoring 

57 and operations support systems, Section |H] describes the response of the CDF 

58 silicon detector to accumulated radiation doses, Section [10] details the perfor- 

59 mance of the silicon detector and the displaced vertex trigger, and Section [TT] 
eo gives a summary. As well as new results, this paper compiles final results on 
ei material dispersed in several conference proceedings produced over the years by 
62 the members of operations team (7- 12|. 



2. Detector Description 

The CDF silicon detector system consisted of three sub-detectors, all with 
barrel geometry: Layer 00 (LOO) d El, the Silicon Vertex detector (SVX- 



II) 1J, [l5j and the Intermediate Silicon Layers (ISL) [16J. Unless otherwise 
stated, detector refers to the CDF silicon detector. The design of the system was 
driven by the goal of providing excellent spatial resolution in the measurement of 
charged-particlc tracks. These measurements were crucial for the reconstruction 
of the displaced secondary vertices and therefore, identification of events with 
bottom-quarks. Figures [5] and [3] present the schematic layout of the CDF silicon 
detector, and Table Q] summarizes some of the basic parameters. The design 
had eight silicon layers to provide tracking which is robust against failure or 
degradation of individual components. 
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Figure 2: Schematic layout of the CDF silicon detectors showing x-y {r-cj>, left) and y-z (r-z, 
right) views. Note that the z axis is compressed for illustration purposes. 

75 The basic structural unit of a sub-detector was a ladder, which consisted of 

76 several silicon microstrip sensors (3 sensors for LOO ladders, 4 in SVX-II ladders 

77 and 6 in ISL ladders) with strip width and multiplicity depending on the layer, 

78 or distance from the beam pipe. The sensors were made from high-resistivity 

79 n-type silicon with a nominal thickness of 300 fim. Sensors in LOO were single- 
so sided, providing r-<p information, while sensors in the other layers were double- 
si sided, providing both r-(f> and r-z information. The sensors in SVX-II layers 0, 

82 1, and 3 used double-metal readout for a 90° strips on the r-z side. The other 

83 double-sided layers used small-angle stereo strips. 

84 The readout was carried out through aluminum strips AC coupled to the 

85 implant strips, which are of p-type for the r-(j) and n-type for the r-z side. A full 

86 ladder was read out from both ends through SVX3D readout chips (described 

87 in Section 13.3.11) mounted on electrical hybrids. These hybrids were located 

88 outside (for LOO) or inside (for SVX-II and ISL ladders) of the tracking volume. 

89 A circuit board called the portcard was located at the periphery of each support 

90 structure or bulkhead and formed an interface with the hybrids and readout 

91 chips with the rest of the data acquisition system (Section [3]). 

92 Layer 00 was a single-sided silicon microstrip detector whose sensors could 

93 be biased to higher voltages than the double-sided sensors. It was mounted on 

94 a carbon fiber support structure which was in turn mounted directly on the 

95 beam pipe, and had an inner radius of 1.15 cm and outer radius of 2.1 cm. 

96 Its main purpose was to improve the track impact parameter resolution which 

97 was otherwise limited by multiple scattering in the additional material of the 

98 SVX-II readout and cooling infrastructure; a secondary purpose was to prolong 

99 CDF silicon detector lifetime by providing a backup to SVX-II layer-0. Layer 

100 00 consisted of one layer and had 72 ladders with 13,000 readout channels in 

101 total. 
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|LOO+SVXII+ISL, xy view] |LOO+SVXII+ISL, yz view] 
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Figure 3: Dimensions, in cm, of the CDF silicon detector system. Shown are x-y and y-z 
views. In the y-z view, each square corresponds to one sensor and each subdetector has been 
displaced along the y-axis for illustration purposes. 



The SVX-II detector was built in three cylindrical barrels each 29 cm long. 
Each barrel contained five layers of double-sided silicon microstrips placed along 
the beam axis, with radial coverage from 2.5 to 10.7 cm. Carbon fiber re- 
inforced Rohacell foam [17j provided support to the ladders, and beryllium 
bulkheads provided additional support and alignment on each end. Therefore 
the detector consisted of six bulkheads (^-segmentation), each with 12 wedges 
(^-segmentation) consisting of 5 layers (r-segmentation). In total, it had 360 
ladders with 405,504 channels in the system. One side of each microstrip sensor 
provided tracking information in the r-<p plane, with strips oriented parallel to 
the beam direction, while the other side had strips oriented either perpendicular 
to the beam axis, providing 90° information, or at an angle of ±1.2° with respect 
to the beam axis, providing small-angle stereo information. Three of the five 
SVX-II layers had 90° sensors, while the remaining two layers had small-angle 
stereo strips, as detailed in Table [TJ The readout chips and electric hybrids were 
mounted on the surface of the SVX-II silicon sensors. SVX-II was read out in a 
strict </)-wedge geometry in order to feed the secondary vertex trigger, described 
in Section f3. 1.1 1 

The ISL was located between SVX-II and the COT drift chamber. It con- 
sisted of one central (|ry| < 1) layer of silicon at a radial position of 22 cm and 
two forward (1 < < 2) layers at 20 cm and 28 cm. Mechanical support for 
the ladders was provided by carbon fiber rings. ISL had 148 double-sided lad- 
ders of 55 cm length each with a total of 303,104 channels. An ISL ladder was 
composed of three microstrip sensors bonded together. Like SVX-II, one side of 
each sensor provided tracking information in the r-cf> plane while the other side 
provided tracking information in the r-z plane with ±1.2° stereo angle. Also 
like SVX-II, the readout chip hybrids were mounted on the sensors. 

Figure [4] gives a historical account versus time (left) and integrated luminos- 
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Table 1: Summary of LOO, SVX-II and ISL basic parameters. 



Name 


Radius (cm) 


Readout 


Manufacturer 


LOO (narrow) 


1.35 


r-(f> 


SGS Thomson, Micron 


LOO (wide) 


1.62 


r-<j) 


Hamamatsu 


SVX LO 


2.54 


r-<j),r-z 


Hamamatsu 


SVX LI 


4.12 


r-(f>,r-z 


Hamamatsu 


SVX L2 


6.52 


r-0,+1.2° 


Micron 


SVX L3 


8.22 


r-<p,r-z 


Hamamatsu 


SVXL4 


10.10 


r-(j),-1.2 


Micron 


ISL L6 Central 


22.00 


r-4>,l.2° 


Hamamatsu 


ISL L6 Fwd/Bwd 


20.00 


r-<f>,1.2° 


Hamamatsu 


ISL L7 Fwd/Bwd 


28.00 


r-(j},l.2° 


Micron 



129 ity (right) of the fraction of detector ladders included in data taking since start 
of commissioning in 2001. Aside from variation during the start-up period, 



—All Good —Bad —Errors —All Good —Bad —Errors 




Year Integrated Luminosity [pb 1 ] 



Figure 4: Fraction of ladders which were powered (black), considered good (green) and bad 
(red) versus time (left) and versus integrated luminosity (right). A ladder is considered good 
if it has less than 1% digital errors. Also shown is the average digital error rate (pink). The 
actual fraction of good ladders is larger as it does not included ladders whose digital errors 
are corrected by the offline reconstruction software. 

130 

131 stable detector operation is observed over the long term data taking period. 

132 The rate of corruption in data transmission out of the detector (referred to as 

133 " digital errors" ) rises with time. Some of this corruption was recovered with of- 

134 nine processing by using knowledge of the data structure to identify and correct 

135 erroneous bits. 
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136 3. The Silicon Detector Data Acquisition System 

137 The data acquisition (DAQ) system of the silicon detector was responsible for 
us reading out and digitizing the charge collected by the 722,432 silicon strips. The 
139 DAQ worked in coordination with the CDF trigger system that selected events 
ho (proton-antiproton collisions) of interest [lj]. The DAQ system was comprised 

141 of radiation-hard readout ASICs mounted on the detector, feeding optical data 

142 links and a chain of VME boards that coordinated the DAQ process and that 

143 collated and processed the data. The unique feature of the DAQ system was 

144 the integration of the silicon detector with the secondary vertex trigger (SVT) 

145 which had never been attempted at a hadron collider. 

we The first half of this section (Sections ED} ED} describes the CDF DAQ 

147 and then details the components of the silicon detector DAQ. The second half 

us (Sections 13.5.11 13. 9p describes the commissioning and operations experience, 

149 which includes the unexpected behaviors of the SVX3D chip, noise from LOO, 

150 and effect of radiation on the DAQ. 

151 3.1. CDF Timing and Trigger 

152 The CDF trigger and DAQ systems were synchronized to the Tevatron 

153 beams. The Tevatron divided the proton and antiproton beams into 3 trains, 

154 separated by 1.4 /is, and each train was composed of 12 bunches separated by 

155 396 ns. In total there were 36 bunches of protons and antiprotons. It took 

156 21 fis to complete one revolution of the Tevatron. The orbits of the proton and 

157 antiproton beams were set to collide every 396 ntQ at the two points where the 

158 CDF and DO detectors were located. 

159 The CDF clock signals were derived from the Tevatron clock system. The 

160 most fundamental clock was derived from the 53 MHz Tevatron radio frequency 

161 (RF) systenH The Tevatron also sent a signal corresponding to the first proton 

162 bunch, with a period of 21 (is (1113 Tevatron RF clock periods) in phase with 

163 the RF clock. Further details on the Tevatron beam structure and clocks are 

164 available in [l8[ and references therein. The fundamental CDF clock was derived 

165 by dividing the Tevatron RF clock by 7, which gave a period of 132 ns and phased 
lee with the first proton bunch clock. Additional clock signals were derived for valid 

167 bunch crossings and the gaps between trains. 

168 At a hadron collider, only a small fraction of events are from interesting 

169 physics processes. The CDF trigger was responsible for identifying these inter- 

170 esting events in real time. CDF employed a three-level trigger system, where 

171 each level used more refined information than the previous level to select events. 

172 The first level (LI) ran synchronously with the CDF clock and had a fixed la- 

173 tency of 5.5 /zs. It reduced the event rate from 1.7 MHz to less than 35 kHz, 

174 and was implemented with custom hardware [l| . When events were selected by 



lr The Tevatron proton-antiproton collision rate was intended to be upgraded to 132ns, but 
this change was not implemented. 

2 The central Tevatron RF system fed the Tevatron's accelerating RF cavities. 



8 



175 LI, the data for non-silicon detectors were stored in one of four buffers, pending 

176 processing by the second level (L2). L2 was an asynchronous trigger, comprised 

177 of dedicated hardware and software, that selected a subset of the LI triggered 

178 events. It reduced the peak rate of accepted events to ^800 Hz. The third level 

179 (L3) was a software trigger that ran a fast version of the offline event recon- 
lso struction on a computer farm using all data from the CDF detector. It selected 
lsi a subset of L2 triggered events for permanent storage at a rate of ~150 Hz. 

182 Overall, the CDF trigger selected ~1 in 11,000 collisions for permanent storage. 

183 3.1.1. SVT 

184 A unique feature of the CDF's L2 trigger was the ability to select events 

185 with a displaced vertex which were characteristic of bottom quark hadron (b- 

186 hadron) decays. This method of selecting hadronic 6-decay events was more 

187 efficient than previous leptonic triggers that relied on the rarer semi-leptonic b- 

188 quark decay. This displaced vertex trigger, known as the Silicon Vertex Trigger 

189 (SVT), significantly increased CDF's yield of 6-hadrons for analysis. 

190 The SVT used data from the CDF silicon detector and COT to perform 

191 precision tracking quickly. Tracks were found by combining information from 

192 the COT-based extremely fast tracker (XFT) [l9( and SVX-II axial layers to 

193 patterns stored in look-up tables. The resulting tracks in the r-<f> plane were 

194 used to calculate the 2D distance (L xy ) of a track pair intersection from the 

195 primary vertex. A key development of the SVT hardware was the custom chip- 

196 based pattern recognition (associative memory). CDF was the first detector at 

197 a hadron collider to implement a disp laced vertex trigger. Further information 

198 on the SVT can be found in [2CH22| and references therein. The demands of 

199 SVT to reconstruct tracks and identify tracks displaced from the interaction 

200 point drove the SVX-II design and led to the wedge and barrel layout, the tight 

201 construction alignment tolerances, and the SVX-II DAQ design that is discussed 

202 in this section. 

203 3.2. CDF Silicon DAQ architecture 

204 SVX-II was designed for SVT, which required a specialized DAQ system to 

205 provide silicon strip data in 20-40 (is. The ISL and LOO that came as extensions 

206 to the CDF silicon detector project inherited the SVX-II DAQ. Thus the SVX- 

207 II DAQ defined the entire CDF silicon detector DAQ. SVT demanded SVX-II 

208 provide data after every L1A. This required deadtimeless readout to guarantee 

209 silicon data was always available. Also, to reduce the time to deliver and pro- 

210 cess the data, only information from silicon strips which collected a significant 

211 amount of charge relative to the nominal noise signal was propagated to SVT. 

212 Therefore the silicon readout volume, and thus readout time, was driven by the 

213 underlying physics processes that drove the occupancy of the detector. It was 

214 only by meeting these design challenges that allowed the combination of SVX-II 

215 and SVT to be integrated into the CDF trigger. 

216 Figure [5] shows a schematic diagram of the silicon DAQ system. The pro- 

217 cess was coordinated by a central controller that interfaced with the trigger 
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218 system, broadcasted commands to the individual ladders, and controlled data 

219 flow through feedback signals from the VME readout buffers. The digitized data 

220 from each ladder was transferred in parallel to the allocated readout buffer. The 

221 data were buffered until a L2 decision arrived and was either transmitted to L3 

222 or discarded. The Silicon DAQ was synchronized to the CDF clock. The elec- 

223 tronics were divided into an on-detector part, mounted directly on the silicon 

224 detector, and an off-detector part, located in VME racks in the CDF collision 

225 hall and in the CDF counting room (Fig. [5]). 



SVX II Wedge - 44 Readout Chips 




Figure 5: A schematic diagram of the silicon detector DAQ system. The SVX-II wedge and 
portcard, at the top half of the figure, were known together as the on-detector electronics as 
they were located directly on the silicon detector. The components, in the lower half of the 
figure, were known as the off-detector electronics. The FIB and FIB fanout were located in 
the CDF collision hall. The VRB, VRB fanout and SRC were in the CDF counting room 
which was located above the CDF collision hall. ISL and LOO DAQ were identical except the 
SVX-II wedge was substituted with a ISL and LOO wedge, respectively. 
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3.3. On-detector Electronics 

The on-detector electronics were responsible for acquiring and digitizing the 
charge from the silicon strips and then transmitting the data to the off-detector 
electronics. The on-detector electronics consisted of the SVX3D readout chip, 
the portcard, and DOIM optical data links that are described below. 



3.3.1. The SVX3D Readout Chip 

The SVX3D ASIC was responsible for acquiring charge from the silicon strips 
and digitizing them. It was a custom, radiation-hard, deadtimeless, 128-channel 
device 23ll24j. capable of recording charge simultaneously from all 128 channels 



every 132 ns and had an 8-bit wide output running at 53 MHz. All 128 chan- 
nels were digitized simultaneously using an 8-bit grey-coded modified Wilkinson 
type analog-to-digital converter (ADC) that included event-by-event dynamic 
pedestal subtraction (DPS). With DPS, it was no longer necessary to read out 
every strip for an offline pedestal subtraction. Therefore the SVX3D could 
implement data reduction logic (sparsification) to remove channels that were 
below a set threshold to reduce further the readout time. In this way, the read- 
out time was dictated by the occupancy of the silicon detector. The SVX3D's 
deadtimeless operation, DPS, and sparsification were essential for SVT. 

The SVX3D was manufactured using the Honeywell CMOS 0.8 /im radiation- 
hard process. From irradiations up to 4MRad with 60 Co sources and 15MRad 
with a 55 MeV proton source, the chip noise in the innermost layer of SVX was 
expected to increase by 17% after Sfb" 1 (3.1MRad) [H]. 

The operations of the chip were divided into an analog front end (FE) and a 
digital back end (BE). The FE was responsible for acquiring charge from the sil- 
icon strips and buffering them into a circular analog pipeline. The BE digitized 
the charge from the pipeline and any channels above a programmed threshold 
were sent to a readout FIFO. The FE and BE were driven independently by the 
CDF clock and Tevatron RF clock, respectively, which ensured the deadtimeless 
operation. 

The chip operation began with an initialization phase which set various 
operational parameters such as the signal polarity, chip identification number 
(chip ID), and readout mode. After the initialization phase, the FE changed to 
the acquisition mode, and the BE alternated between digitization and readout, 
until a new initialization was performed. 

Each FE channel consisted of a charge integrator coupled to a 47 stage 
circular analog capacitor pipeline; 42 pipeline cells were allocated for the LI 
latency, four for L2 buffers, and one reserved to measure the pipeline pedestal. 
Every CDF clock cycle, the FE charge integrator acquired charge from the silicon 
strips and transferred it to an empty cell in the analog pipeline. Whenever a 
LI decision to keep an event arrived, the appropriate pipeline cell was marked 
and skipped over until it was digitized. Under normal operations, up to four 
pipeline cells could be marked. 

A marked analog pipeline cell (capacitor) was digitized by the BE. The volt- 
age across the marked pipeline capacitor, subsequently referred to as the strip 
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270 voltage, went to the input of a comparator. The other input of the compara- 

271 tor was a voltage ramp shared by each channel's comparator. At the start of 

272 digitization, the common voltage ramp started and an 8-bit grey-coded counter 

273 started to increment. A channel's comparator changed state as soon as the volt- 

274 age ramp was larger than its strip voltage, which triggered a latch to store the 

275 current value of the 8-bit counter. Therefore channels with smaller strip volt- 

276 ages would trigger their latches earlier. The common pedestal subtraction was 

277 implemented by delaying the counter until the first 33 comparators had changed 

278 state. The choice of 33 channels came from a study to optimize pedestal removal 

279 with signal efficiency. DPS implicitly assumes a constant chip-wide pedestaH 

280 and insulates the strip charge measurement from environmental noise. 

281 The SVX3D implemented three modes of data reduction: read out of strips 

282 above a set threshold (sparse), sparse strips and adjacent below-threshold strips 

283 (nearest neighbor, NN), and no data reduction (read-all). Sparse mode had the 

284 smallest data volume and therefore smallest read out time but NN was chosen as 

285 the additional information proved useful for correcting single bit errors. Except 

286 in the read all mode, the data volume was driven by the underlying physics that 

287 drove the occupancy of the silicon strips. 

288 Multiple chips were chained together to read out a single silicon sensor; two 

289 for the innermost (narrowest) layer and up to 16 for the outermost (widest) 

290 layer. Chip initialization and commands were transferred serially from the first 

291 chip in the chain to the last chip in the chain. The first strip of the first chip 

292 and last strip of the last chip in the chip chain were always read. Data from 

293 each chip in the chip chain were transmitted one-by-one on a common data bus. 

294 The data from each chip included the chip ID and the SVX3D channel number 

295 of each read strip. There were 5644 SVX3D readout chips in the CDF silicon 

296 detector which dissipated approximately 3 kW of power, thus active cooling was 

297 essential for stable operations (Section [5]). 

3.3.2. Portcard 

299 The portcard was the interface between the on-detector and off-detector 

300 electronics. It relayed SVX3D commands and trigger signals from the DAQ 

301 boards and passed on power to bias the silicon sensors and chip chains from 

302 the power supplies (Section d]). Data from the chips chains passed through the 

303 portcard onto the optical data links (DOIMs) (Section [3X3]). As the portcards 

304 were located within the silicon detector, they were designed to have low mass to 

305 minimize the radiation length, to withstand radiation doses of up to ~200kRad 

306 (^lOfb" 1 ), and with high heat transfer capability [26| , 

307 3.3.3. Optical data link: DOIM 

308 The Dense Optical Interface Module (DOIM) was the optical data link 

309 used to transmit data from a chip chain to the off-detector electronics [9( . Each 



3 This assumption was not true for LOO and DPS was turned off for LOO readout (Sec- 
tion [32J. 
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DOIM had a transmitter unit (TX) located on the Portcard and a receiver unit 
(RX) in an off-detector VME-transition module (Fig. EJ). The TX and RX were 
connected by optical fibers. Each DOIM was capable of transmitting 8-bit wide 
data at 53 MHz with an error rate of less than 1 in 10 12 words. 

The DOIM TX housed twelve 1550 nm InGaAsP edge emitting lasers in a 
single package. Only 9 of the 12 were used: 8 to transmit data and one as a 
data-strobe. The DOIM RX was a InGaAsP/InP PIN diode array. It received 
the optical signal from the TX and converted it back to an electrical signal. 
The DOIM TX was tested for radiation hardness with 30 MeV, 63 MeV, and 
200 MeV protons with radiation doses up to 2MRad. The light degradation 



was measured to be 10% after 200kRads 27 



3.4- Off- detector Electronics 

The off-detector electronics were responsible for coordinating the silicon 
DAQ process as well as processing and packaging the digitized silicon strip data 
for the SVT and the CDF DAQ. 

The off-detector electronics were housed in eighteen 9U VME crates using 
VME64 (28|. Of these eighteen crates, eight were located in the CDF collision 
hall close to the CDF detector, while the other ten were located in a counting 
room in the CDF assembly building (Fig. [5]) . The main difference between the 
two sets crates was the use of two different custom J3 backplanes to accommo- 
date different types of boards. All together, there were 164 VME boards. 

Because the data from the SVX3D chip chains were sparsified, the first 
strip of the first chip and last strip of the last chip of the chip chain were 
always reported in order to identify the start and end, respectively, of the chip 
chain data stream. The off-dctcctor electronics appended its own unique header 
to these data. The combination of the header, chip ID, and SVX3D channel 
encoded the unique location within the silicon detector of each digitized charge. 

3.4-1- Silicon Readout Controller (SRC) 

The SRC was the master controller of the silicon detector DAQ and also 
acted as the interface to the CDF DAQ and trigger systems. The SRC was 
housed in a rack in the CDF counting room, which was also shared with the VME 
readout buffers (VRB), and received the clock and the beam structure from the 
central CDF clock fanout. It communicated with the Trigger Supervisor (TS), 
which was the central CDF trigger processor. The SRC also provided the central 
clock to the entire CDF silicon detector, which was kept in sync with the CDF 
clock using a phase locked loop (PLL). The SRC commands, clock, and trigger 
signals were transmitted by the SRC via a Transition Module (SRCTM) to the 
Fiber Interface Board (FIB) crates in the CDF collision hall using a GLINK [2^] 
optical link running at 53 MHz. 

The Silicon DAQ was originally designed to be driven by a single SRC. 
However, the need to read out all channels of LOO every event (Section 13 . 7[) 
required two SRCs, one to drive SVX-II and another to drive ISL and LOO 
(Section 13.7. 1| . Implementation of the second SRC also helped mitigate the 
wirebond resonance problem (Section [ 
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354 3.4-2. Fiber Interface Board (FIB) 

355 The Fiber Interface boards (FIB) were housed in eight crates located in the 

356 four corners of the CDF collision hall. SVX-II and ISL/LOO had four FIB crates 

357 each. The signals from the SRC were received by a FIB Fanout (FFO) board 

358 in each FIB crate and distributed to the FIBs in its crate via a custom J3 

359 backplane. Each FIB communicated with two portcards via a FIB Transition 

360 Module (FTM) on the backside of the FIB crate. It converted the high-level 

361 SRC commands into a sequence of instructions suitable for the SVX3D chip 

362 chains, which were sent with clock and trigger signals to the two portcards. 

363 The FTMs also housed the DOIM RX that received the digitized SVX3D data, 

364 which were passed to the FIB. The FIB formatted the data stream, appended 

365 its own unique header, and sent the data on four GLINKs to the VME Readout 

366 Buffers (VRB) with a copy sent to the SVT through optical splitters. 

367 3.4.3. VME Readout Buffer (VRB) 

368 The VME Readout Buffers (VRB) were located in the VRB crates in the 

369 CDF counting room. Two VRB crates also housed the two SRCs. The VRB 

370 buffered the data from the FIBs until a L2 decision was made by the CDF trigger 

371 system, upon which the event was moved to the output buffer and was collected 

372 by the Event Building system using the VME Bus. The communication between 

373 the VRBs and the SRC was handled by the VRB fanout system, which enabled 

374 the SRC to manage the buffer provided by the VRBs. 

375 The data from each VRB crate were transferred in parallel to the event 

376 builder, which combined segments from the crates into an event record which 

377 was then passed to L3. The SVX-II had 6 VRB crates that corresponded to the 

378 6 SVX-II bulkheads. The ISL and LOO originally had two and one VRB crates, 

379 respectively. To cope with high instantaneous luminosity (above 10 32 cm" 2 ^ 1 ), 

380 it was necessary to reduce the size of the data segments arriving from the VRB 

381 crates. In 2006, the ISL and LOO VRB crates were mixed and an additional 

382 VRB crate was added fSection l3.7.2[) . 

383 3.5. DAQ Commissioning 

384 Prior to installation, the VME based hardware and onboard detector elec- 

385 tronics were thoroughly tested through the use of test stands and data emula- 

386 tion at various levels to verify the functionality and robustness of these systems. 

387 However, due to time constraints, there was limited ability to test the two sys- 

388 terns together after installation. Together with the unforeseen consequences of 

389 the environment in the collision hall, this led to several problems emerging in the 

390 course of the first few years of operation that required immediate attention to 

391 alleviate data corruption and potential damage to the detector. The wirebond 

392 resonance and LOO noise problems were severe and are described separately in 

393 sections 13.61 and 15771 respectively. It took from 2001 to 2003 to fully commission 

394 the silicon detector. 
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395 3.5.1. SVX3D Commissioning 

396 The SVX3D chip was thoroughly tested during its development. But a 

397 number of unexpected behaviors, listed below, were encountered during com- 

398 missioning. The chip would latch to a state where the chip current increased 

399 until it exceeded the power supply safety limit and forced a power supply shut 

400 down (trip). These behaviors were circumvented by modifying the SRC, VRB 

401 and FIB firmware. 

402 Abort Digitize. The SVX3D chip had a feature to abort digitization before 

403 completion if L2 had already rejected the event. However this feature made 

404 the chip enter the high current state and trip. The SRC and FIB firmware 

405 were modified to allow the SVX3D chip to always complete digitization which 

406 stopped these failures. 

407 Fifth LI accept. The SVX3D chip could accommodate up to four L1A signals 

408 without releasing a cell in the analog pipeline (Section 13.3.11) . If a fifth LI A 

409 arrived before a pipeline cell was released, the chip would transition to cither 

410 read-all mode or suppress all readout. The SRC firmware kept track of the 
4n number of unreleased pipeline cells, and was modified not to send the fifth LlA 

412 to the chips, and instead to send an error signal back to the CDF DAQ. This 

413 error signal forced a silicon CDF DAQ reset and re-synchronization. 

414 Keep- Alive. At least every 270 /is, a command had to be sent to the chip chains 

415 to prevent chips from entering the high current state and tripping. The SRC 

416 firmware was updated to send these keep alive signals every 270 /is in the absence 

417 of any commands. But the SRC state machine was driven by the CDF clock 

418 and any glitches or interruptions of this clock would also delay or interrupt 

419 the delivery of these keep-alive signals, which would result in large portions of 

420 the chips tripping off. Given the sensitivity of the silicon detector to any clock 

421 glitches, administrative procedures were implemented requiring permission from 

422 either the CDF silicon detector project leader or the head of CDF detector 

423 operations before work was done on on the CDF clock or Tevatron clock. 

424 AVDD2 Errors. There was a class of unrecoverable failures that affected 6% 

425 of the SVX3D chips and could be reproduced only by disabling one of SVX3D's 

426 analog voltage lines (AVDD2). The observed symptoms were loss of commu- 

427 nication with the FE, an increase in the SVX3D's BE current, and loss of 

428 communication to chips beyond the affected chip. This class of failure typically 

429 occurred after a beam incident (Section [6]) or a large temperature change, such 

430 as a cooling system failure. This type of failure became infrequent after 2003 

431 (Fig. [7] and [8J when operation procedures during shutdowns were changed to 

432 minimize thermal cycles, coincident with a sharp decline in the frequency and 

433 severity of beam incidents. 

434 3.5.2. DAQ Board Enhancements 

435 commissioning of the silicon detector, the SRC, VRB and FIB firmware were 

436 extended to circumvent the unexpected behaviors of the SVX3D readout chip, 
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437 which are documented in section 13.5.11 In addition, minor problems appeared 

438 when the off-detector and on-detector components of the system were integrated. 

439 A few of these issues are described in detail to illustrate the type of problems 

440 encountered and solutions implemented. 

441 Data was lost due to failures in the transmission of the clock signal from 

442 the FIB/FTM to the portcard. Electronic components on the fiber interface 

443 input of the FIB were replaced to increase the tolerance of varying duty cycles 

444 on a signal that carried the clock information. The clock rate was known, and 

445 failures in the transmission of its signal were overcome by providing an identical 

446 backup clock signal. The firmware was also updated to increase the allowed 

447 width of the front-end clock from about 28 ns to about 34 ns in order to avoid 

448 the loss of charge collection due to inadequate integration time. 

449 As the readout was data driven with no fixed length, the FIB used the last 

450 channel of the last chip in the chain to identify the end of a chip chain's data 

451 stream. Failure to detect this, which could be caused by the chip, DOIM, or FIB 

452 error, could potentially make the FIB wait for an indefinite amount of time. A 

453 timeout was added to the FIB to terminate readout and append an error code 

454 to the data stream. 

455 Data were also initially lost to a race condition in the data concatenation 

456 algorithm at the VRB level. This condition shifted every other 4 bits in the 

457 data stream by 8 bits, leading to data corruption and the loss of events at the 

458 1% level. Once the systematic shift was distinguished from random corruption, 

459 the VRB firmware was modified to eliminate this source of data loss. 

460 3.6. Wirebond Resonance: Spontaneous loss of r-z sides in the double-sided 

461 ladders 

462 Shortly after the beginning of data taking operations in 2002, 4% of the 

463 r-z side of SVX-II ladders were spontaneously lost during operations. In the 

464 SVX-II, the r-<j) and r-z side hybrids of the ladders were connected with a set 

465 of wire bonds, known as the jumper. The jumper was perpendicular to the 

466 1.4 T magnetic field produced by the CDF solenoid (Section [TJ). On every 

467 readout sequence of the chips, a varying current flowed through the jumpers, 

468 which resulted in a Lorentz force that induced a kick on the jumpers. This 

469 process usually did not lead to a resonant condition, as the readout commands 

470 were typically randomly spaced. However if the readout commands came at a 

471 fixed frequency, a resonant Lorentz force could cause the wire to break from 

472 mechanical fatigue. It had been shown that some resonant frequencies of the 

473 jumpers were in the 10 kHz range (which exactly matched the CDF L1A trigger 

474 rate under certain conditions) and only a few kicks were necessary to excite the 

475 jumpers These resonant readout conditions would arise when there were 

476 synchronous LlAs from calibration triggers, faults in the trigger hardware, and 

477 ladders with large and fixed length readout. The silicon detector was removed 

478 from all calibration triggers and faulty trigger hardware was replaced. LOO had 

479 large fixed readout, discussed in section 13771 and its separation from the SVX-II 

480 readout (Section 13. 7. 1|) was necessary to mitigate the resonances. 
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481 3.6.1. Operational Mitigation 

482 During the initial investigation, it was understood that the damage was 

483 correlated with the LI trigger rate. The maximum LI trigger rate was set to only 

484 1 2 kHz, where the typical peak rate was about 20 kHz. A limit was implemented 

485 in the DAQ software, called the trigger handbrake, that would halt data taking 

486 if the four-second-average of the LI trigger rate exceeded the maximum rate. 

487 After the wirebond resonance was discovered and the Ghostbuster protection 

488 system was commissioned, the maximum rate was raised to 35 kHz. In addition, 

489 an administrative procedure required that every change to the trigger system 

490 was tested without the silicon detector and signed-off by the silicon operation 

491 group. 

492 3.6.2. Ghostbuster protection system 

493 Given that it only took a few kicks at a ~ 10 kHz resonance frequency to 

494 excite a resonance, the Ghostbuster [30], already developed for SVT, was re- 

495 programmed to detect the onset of a resonance condition within ~1 ms. The 

496 Ghostbuster paused data taking as soon as a series of synchronous readout com- 

497 mands had been detected. The FFO was modified to send readout commands to 

498 the Ghostbuster. The addition of the Ghostbuster was essential to allow CDF 

499 and the silicon detector to acquire data at the highest LI rates. Without the 
soo development of this board, the CDF physics program would have been severely 

501 limited. After the introduction of the hardware protection system, losses of the 

502 r-z side in ladders due to resonant conditions were eliminated, except for two 

503 cases in 2005 and 2007 (Fig. [7]). After the commissioning of the Ghostbuster, 

504 the trigger handbrake remained as a redundant limit on the L1A rate. 

505 The Ghostbuster algorithm paused data taking when it appeared that the 

506 timing of readout commands was within a narrow frequency band. The Ghost- 

507 buster recorded the time interval between successive readout commands. A 

508 difference in successive intervals of less than 1 fj,s was counted as a tick. If the 

509 difference in successive interval lengths was greater than 1 /j,s, the tick counter 

510 was reset. A resonance error was declared when the tick counter reached a preset 
5n threshold, typically set to 11. The threshold value was initially determined from 

512 a Monte Carlo simulation of the DAQ, and tuned to running conditions when 

513 necessary. There was always a non-negligible chance that a set of consecutive 

514 random LI triggers would look like a resonance, and the threshold set point was 

515 a compromise between detector safety and limiting false resonance alarms. 

516 Figure[6]shows the the number of resonances per week detected by the Ghost- 

517 buster during all of Tevatron Run II. The spikes in the number of resonances 
sis per week were mostly caused by faulty trigger hardware or long readout times 

519 for ladders. The typical rate, neglecting those originating from faulty trigger 

520 hardware, was about 10 resonance errors per week (1.4 per day) consistent with 

521 stochastic operation. 

522 As mentioned previously, any ladders with long fixed readout would exac- 

523 erbate the likelihood of resonances at some LI rates. The readout time could 

524 increase from corruption of the pedestal subtraction algorithm or if the noise 

525 level had grown. In both cases, the ladder readout time is no longer dictated by 
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Figure 6: A historical account of the number of resonance per week detected by the Ghost- 
buster. The maximum number of ticks, which triggered the Ghostbuster resonance detection, 
varied from 10 to 13 ticks during Run II, and eventually settled at 11. The spikes in the 
resonance rate are mostly due to faulty trigger hardware. The typical resonance rate was 10 
per week (1.4 per day). 



526 detector occupancy but rather ladder noise. This could lead to approximately 

527 fixed long readout times. Also some chips in a chip chain were switched to 

528 read-all mode to fix some errors, which also increased the readout time. During 

529 2009-2010, the noise level in a handful of ladders had grown large enough, and 

530 consistent enough in length, that the number of resonances increased and grad- 

531 ually forced the peak LI trigger rate to be limited to 25 kHz. The problematic 

532 noise growth was due to malfunctions in the chip or the sensor itself, and was 

533 several times larger than the noise growth due to radiation damage observed in 

534 most ladders. The noise was often isolated to a few chips of the chain, and by 

535 increasing the NN sparsification threshold of the affected chips, the noise was 

536 suppressed without compromising the data from other chips in the ladder. With 

537 this noise suppressed, the peak LI A rate was increased to 32 kHz counts during 

538 normal data-taking, without creating resonances. 

539 3.7. LOO Noise 

540 LOO was included to improve the precision of measuring displaced vertices 

541 that was essential for the discovery for B s oscillations |3l|. Unlike SVX-II 

542 and ISL, the LOO readout chips were not mounted directly on the LOO sensors to 

543 minimize the amount of material and so reduce the effects of multiple scattering. 

544 Instead, a fine-pitched cable connected the sensors to the readout chips. 
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545 After LOO installation, significant noise was observed on the LOO readout that 

546 manifested as large pedestals that varied across a chip and with each event, with 

547 the largest variation at the edges of the readout cables. These pedestals could 

548 not be removed by DPS. An investigation concluded the noise was picked up by 

549 the fine-pitched readout cables [f|. 

550 LOO was forced to operate in read-all mode and the pedestals were removed 

551 by an offline event-by-event correction. During the offline data processing, the 

552 recorded charge across a chip was fit to Chebyshev polynomials to extract the 

553 pedestal. Tests using simulation were performed to check for biases from fitting 

554 and none were found 0. 

555 3.7.1. Two-SRC Mode 

556 The original silicon DAQ read out SVX-II, ISL and LOO together. A conse- 

557 quence of operating LOO in read-all mode was that it had the largest data volume 

558 and was fixed length, which exacerbated the wirebond resonances fSection l3.6|) . 

559 As LOO and ISL were not used by SVT, they were separated from SVX-II DAQ 
sec and read out by a separate SRC after a L2 accept. This improved the readout 

561 time and also mitigated the wirebond resonances. 

562 3.7.2. Load Balancing 

563 During Tevatron Run II, the peak instantaneous luminosity increased from 

564 50 x 10 30 cm _2 s" 1 to 400 x 10 30 cm _2 s _1 and this had two consequences: higher 

565 detector occupancy and an increased trigger rate. Without improvements to 
see CDF's trigger and DAQ, CDF would not have been able to operate at this 

567 high instantaneously luminosity. In particular, some data segments arriving at 

568 the event builder were significantly larger than others, and the event builder 

569 performance was limited by the largest of these data segments. To mitigate this 

570 effect, the number of VRB crates was expanded for both silicon and non-silicon 

571 systems, and data rates were equalized across the crates. 

572 As described in Section [3~4l data from the CDF silicon detector was buffered 

573 on VRBs until a L2 decision arrived. Each VRB crate was read out in parallel 

574 and the total readout time was dictated by the LOO VRB crate as it had the 

575 largest data volume. The LOO VRB crate had a fixed event size of 28 kB per 

576 event as a consequence of its read all mode (Section [3~7)l . 

577 The readout time was reduced by mixing ISL and LOO VRBs to balance the 

578 data volume across VRB crates, thus reducing the peak data volume per VRB 

579 crate. A configuration was found and implemented during the 2006 Tevatron 

580 shutdown. The maximum event size of a single crate reduced from 28 kB to 

581 20 kB. 

582 Despite the success of the load balancing, the increased instantaneous lu- 

583 minosity of the Tevatron forced another re-optimization of the CDF DAQ. For 

584 the silicon DAQ, balancing the load across VRB crates was no longer sufficient. 

585 An additional VRB crate was added to the L00/ISL readout and remained in 
sse operation until the end of Run II. 
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587 3.8. Operational Experiences and Improvements 

sss The commissioning of the CDF silicon detector was completed at the start of 

589 2003 and the detector was included safely in normal data taking. This section de- 

590 scribes the routine day-to-day problems that persisted to the end of Run II. The 

591 issues that affected the silicon DAQ on a daily basis were broadly categorized 

592 as effects from single-event-upset and bit errors. Procedures were developed to 

593 resolve common problems but required constant vigilance by the detector op- 

594 erations shift crew. Another issue, which persisted, was the full-detector trips, 

595 where either subdetectors, or the complete silicon detector would switch off. Al- 

596 though far rarer, typically four times per year, it took 45-60 minutes to recover 

597 and resume data taking. Another large component of detector operation was 

598 the daily maintenance of the 580 ladders and 5644 readout chips. With such a 

599 large number of components, at least one readout chip and/or ladder required 

600 some daily adjustment. 

601 3.8.1. Impact of ionizing radiation 

602 A sizable fraction of the DAQ system was installed in the CDF collision 

603 hall, which was subjected to radiation from the Tevatron's colliding beams. As 

604 a result, the majority of electronics failure were due to radiation induced single 
60s event upsets (SEU). A SEU is a change of state caused by radiation striking a 

606 sensitive component in an electronic device. The change of state is a result of 

607 the free charge created by ionization in or close to an important logic element 
60s or memory bit. 

609 During Run II, the Tevatron substantially reduced radiation rates, thereby 

610 reducing the radiation induced failures. At the end of Run II, the rates were 
en so low that they fell below the detectable threshold of the radiation monitoring 
612 counters during the course of a Tevatron store. Also, the introduction of the 
en Silicon-Autorecovery (SAR) in 2008 automated the detection and recovery of 
6« these SEU failures of the DAQ and the power supplies. It reduced interruptions 

615 to data taking from 10-20 minutes to less than five minutes. 

616 Reinitialization of Chip Chains. During data taking, the current consumed by 

617 the analog FE of a SVX3D chip chain (Section I3.3.1[) could spontaneously drop 

618 by 80-100 mA, implying one chip in the chain was not recording any data. This 

619 typically occurred at rate of once or twice a day. A program dedicated to 

620 monitoring the power supplies sent an alarm to the DAQ if such a drop in chip 

621 current was detected, and the data taking was paused for less than a minute 

622 to reinitialize the chip chain. Overall, this only had a minor impact on data 

623 taking. 

624 FIB Bit Errors and FIB FPGA Burnout. Each FIB had 16 FPGAs, and thus 

625 a higher rate per board of SEU damage than the other VME modules. Data 

626 corruption errors were resolved by reloading the FPGA programs of the affected 

627 FIB, which typically occurred 1-2 times a day. On rare occasions (3-4 per year), 

628 one of the FPGAs would enter a high current state and blow a fuse on the 

629 board. In a majority of cases the affected FPGA had to be replaced, and in the 

630 remaining cases, a reprogramming of its firmware was needed. 
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631 3.8.2. Bit Errors in the Data stream 

632 As the silicon DAQ is data driven, the data format had to be self-describing 

633 to identify not only the amount of charge but also the location where it was 

634 recorded. Therefore any corruption of the data implied more than just errors in 

635 digitization. During operations, several sources of bit errors in the data stream 

636 were detected and immediately addressed. While some errors, especially those 

637 from on-detector components, could not be repaired, many could be corrected in 
ess the offline reconstruction. Operating the SVX3D readout in nearest-neighbor 

639 mode guaranteed at least 3 consecutive strips would be read, and the crror- 

640 correction algorithm could exploit this feature to identify and correct single-bit 

641 errors. 

642 Bit errors in the optical links. One common instance of bit errors in the data 

643 stream was in the DOIM system. At the start of Tevatron Run II, most DOIM 

644 bit errors were traced to bad electrical contacts in the sockets that held the 

645 RX in the FTM. Gold plating the pins of those devices to establish a better 

646 connection with the sockets eliminated this source of bit errors. 

64? During Tevatron Run II, the typical DOIM bit errors manifested as bits 

648 that were stuck low or high in the data stream, which corresponded to either 

649 a malfunctioning RX unit or TX unit. Faulty RX units were accessible and 

650 replaced from a pool of spares when necessary. Faulty TX units were inaccessible 

651 and thus irreplaceable. Some TX failures could be recovered by adjusting power 

652 supply settings to tune the TX unit light output. Another class of TX-rclatcd 

653 errors were linked to bad connections in a circuit board which served as the 

654 interface between the sensors and the power supplies. These boards were located 
ess just outside the tracking volume and were accessible only when the Tevatron 
656 was shut down for at least a week. In these instances a borescope and a custom 
es? tool were used to push the circuit board back into place to re-establish the 
ess electrical connection. 

659 Bit errors in the Readout boards. The FIB occasionally caused bit errors in the 

660 data stream, most commonly due to SEUs and component failures. SEU related 

661 bit errors were resolved by reloading the programs to the FPGAs on the boards. 

662 FIBs boards with failed components were replaced. On rare occasions (less than 

663 once per year), VRB boards gave bit errors that were traced to component or 

664 printed circuit boards failures and were replaced. 

ees 3.8.3. Full Detector trips 

eee There were several incidents in which most or all SVX3D chips in the silicon 

ee? detector would go into a high current state leading to power supply trips. These 

ees trips occurred in some or all of the SVX-II, ISL, and LOO sub-detectors and 

669 occurred more frequently in the winter season. On average, it took about 45-60 

670 minutes to recover from these incidents. 

671 Only some of the sources of these trips were reproducible and the remainder 

672 were hard to diagnose and resolve due to their rarity. While the origin of all 

673 these trips was unknown, many potential causes were identified and eliminated 
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674 as sources of the problem. It was observed several times that personnel working 

675 near the electronics area could induce this problem, suggesting that loose or 

676 corroded contacts may have been a source; the re-seating and replacing of many 

677 key components proved inconclusive. It was also suspected that differences in 

678 grounding levels between the racks could generate this problem, but no evidence 

679 of a bad ground was found. Another of these sources was the corruption of the 

680 clock signal and consequent lack of keep-alive commands sent to the chip, as 

681 detailed in Section 13.5.11 The underlying reasons for corrupted clock inputs 

682 were not clear, and many full detector trips did not show any indication of 

683 clock corruption. 

684 It was found that some of these trips did occur in coincidence with a high- 
ess voltage power supply trip of one particular muon detector chamber. The muon 
ese detector chamber was powered with 3500 V and had a current draw of about 1 
687 mA. The trip of its power supply during these incidents was attributed to arcing 
ess between the high-voltage lead and the chamber ground. The hypothesis that 

689 an arc in a completely unrelated subsystem would give rise to a massive power 

690 trip in the silicon detector was tested by inducing an arc in the muon detector 

691 chamber and observing the behavior of the silicon detector. The arc was forced 

692 by closing in the high- voltage lead to the ground until a spark was generated, 

693 and full subdetector trips in the silicon detector were reliably reproduced each 

694 time the spark was induced. The mechanism was believed to be electromagnetic 

695 pickup between the muon chamber high voltage distribution cables and the clock 

696 crate during the occurrence of the spark. The electromagnetic field bursts were 

697 observed by carefully-placed coils in the surroundings of the clock crate. The 

698 pickup induced a change in the ground level of the clock signal, which exceeded 

699 the specification for the silicon electronics, resulting in an effective lack of clock 

700 signal during a period of time of about a few microseconds; thus no keep-alives 

701 were sent during this clock interruption. This problem was solved by reducing 

702 the voltage applied to the defective muon detector chamber to 3200 V. 

703 3. 9. Summary of DA Q Performance 

704 The CDF silicon DAQ was incredibly complex in order to meet the chal- 

705 lenges of providing data to SVT. Its performance during Tevatron Run II was 

706 defined by three major phases: commissioning (2001-2003), steady operations 

707 (2004-2008), end of Run II (2009-2011). The long commissioning period was 
70s directly due to the large number of unexpected problems that were apparent 

709 only after installation of the silicon detector. The challenge of tackling these 

710 problems simultaneously (in addition to a number of non-DAQ issues), while 
7n attempting to take data simultaneously was considerable. However after this 

712 commissioning period, the silicon DAQ entered a stable period and efforts to 

713 optimize the performance and operations were carried out, in addition to daily 

714 maintenance of the DAQ components. From 2009, the dwindling pool of working 

715 spare components and the effects of radiation were taking their toll. 

716 Figures [7] and [8] are historical records of the different types of errors accrued 

717 by the DAQ electronics during Tevatron Run II, expressed as the fraction of 
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718 bad readout chips. Figure [7] shows the SVX-II r-cj) and r-z sides separately. The 

719 definition of the different errors are: 



720 • AVDD2: SVX3D errors diagnosed as AVDD2 type errors (Section |3XT|) 

721 • SVX3D: SVX3D errors that are not AVDD2 type error (Section EXT) 

722 • Detector: Faults which originate in the silicon sensor 

723 • Optical: Errors which are from DOIM TX or RX 

724 • Jumper: Ladders whose r-z side was lost from wirebond resonances (Sec- 

725 tion 13. 6j) 



726 • Cooling: Ladders turned off due to lack of ISL cooling (Section 15.2. 1[) 

727 • Hardware: Error and faults which do not match any of the categories 

728 defined above 

729 During the commissioning phase, the number of bad chips (ladder) grew as 

730 the different problem manifested, then stabilized in 2003 (steady state running) . 

731 The r-z plots in Fig. [7] shows that there were only 4 additional jumper failures 

732 after the inclusion of the Ghostbuster and none of these was an immediate 

733 consequence of beam incidents or resonant conditions. After the inclusion of the 

734 Ghostbuster, data was taken safely and reliably with peak LI trigger rates in 

735 excess of 25 kHz - essential for CDF physics. The failure rate of chips after 2003 

736 is far lower compared to the commissioning period. Most of the different failure 

737 categories stabilized. From 2009, the chronic effects of radiation damage and 

738 aging were starting to take their toll on the silicon detector. The failures from 

739 SVX3D and optical were steadily increasing and half the total radiation dose 

740 was delivered between 2009-2011. With the increased radiation, components 

741 started failing more often, shrinking the pool of spare components. These plots 

742 also highlight other problems of the silicon detector that affected the DAQ. 

743 Figure [5] shows a large rise and fall in the fraction of bad ISL chips during 

744 2003 due to blocked cooling line and its eventual clearance; this is discussed in 

745 further detail in section [5j 

At the end of Run II, 84% (73%) of all SVX-II r-<j) [r-z) and 89% of all ISL 

747 SVX3D chips continued to function without error and did not compromise the 

748 silicon detector tracking performance. This is an impressive feat as the CDF 

749 silicon was designed to be replaced after the first 2-3 fb _1 , about three years. 

750 It survived four times the radiation dose and lasted three times longer than the 

751 original design. 

752 4. Power Supplies 

753 The CDF silicon detector used power supply modules manufactured by 

754 CAEN. A total of 114 custom modules were housed in 16 SY527 mainframe 

755 crates located in the corners of the CDF collision hall. The crates were elevated 
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Figure 7: A historical account of the different errors and faults accrued by the SVX-II r-tj> 
(top) and r-z (bottom) SVX3D chips during Tevatron Run II. Each error category is defined 
in Section I3.9I At the end of Tevatron Run II, 84% r-<f> and 73% r-z SVX chips were still 
operating without error. 

756 2-7 meters off the floor due to space constraints in the collision hall. This lo- 

757 cation had a distinct disadvantage; the supplies and crates were continuously 

758 exposed to radiation, which not only shortened the life of many internal elec- 

759 tronic components, but also resulted in single event upsets that required a crate 

760 reset and in single event burnouts that necessitated additional hardware pro- 

761 tection for the detector. Investigation or replacement of a problematic power 

762 supply required access to a radiation controlled area for 1-2 hours. 

763 J^.l. System Overview 

764 One power supply module provided low voltages (2 V and 5 V) to the port- 

765 card, low voltages (5-8 V) to the SVX3D chip chains, and high voltage (up to 
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Figure 8: A historical account of the different errors and faults accrued by the ISL SVX3D 
chips during Tevatron Run II. Each error category is defined in Section I3.9I At the end of 
Tevatron Run II, 89% of ISL SVX3D chips were still operating without error. 



766 500 V) to bias the sensors of one wedge of the silicon detector. The low voltages 

767 were set via potentiometers on the side of the supply, while the high voltage was 

768 set via software on the SY527 crate. All channels had a maximum voltage set- 

769 ting. If the channel voltage exceeded its maximum, an "overvoltage" error was 

770 met and the power supply cut power to the affected channel. These limits were 

771 also set via potentiometers on the modules. Any adjustments to the low voltage 

772 or the maximum voltage settings had to be done with the supply inserted into 

773 the crate and the crate powered. Because of this need and the location of the 

774 potentiometers, changing these settings were not possible in the collision hall, 

775 rather it had to be done on a test stand, using a specially modified crate. 

776 The SY527 crate had an RS-232 interface for connection to a computer, as 

777 well as a Lemo input to reset the crate remotely. In addition, the crate had a 

778 proprietary serial communications port, which connected to a V288 high speed 

779 CAENet VME Controller. The V288 connected to another controller card in the 

780 same crate which had an ethernet port for external communications. A VME 

781 crate was used to communicate with all the crates in the collision hall this way, 

782 and a Java program provided a graphical interface of the power supply controls. 

783 A PC monitored and logged parameters, including power supply voltages, cur- 

784 rents, and the time of last communication for each power supply crate. This 

785 PC logged parameters for the cooling system as well (see Section [SJ. 

786 Decreasing Low Voltages 

787 In 2005, after over 4 years of data taking, low voltage channels of several 

788 power supplies were found to drift erratically from their nominal values. Specifi- 

789 cally, filter capacitors in a particular regulator circuit of each low voltage channel 

790 gradually lost their capacitance, causing the low voltage supplied by the circuit 
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791 to drop over time, typically a tenth of a volt over the course of three months. 

792 Left unchecked, this could cause the readout electronics to stop working. 

793 The solution was to replace all of the filter capacitors (32 in total) on the 

794 supply. This repair was done on-site at FNAL, and typically only took a few 

795 hours. The swaps were usually not urgent (the readout electronics would still 

796 work at 0.5 V below nominal), and were done when another problem in the 
detector required an access to the collision hall. Figure [9] shows the analog low 
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Figure 9: An analog low voltage line powering a series of SVX3D chips of an SVX-II ladder 
is seen dropping gradually over time. The vertical dashed line marks the replacement of the 
faulty power supply. 

797 

798 voltage channel of an SVX-II ladder drifting over a period of two years. The 

799 vertical dashed line marks the time when the faulty power supply was replaced, 
soo restoring the low voltage to its nominal value. The gaps in the plot correspond 
soi to Tcvatron maintenance shutdowns. During the 2007 (2009) shutdown 47 (21) 

802 supplies out of a total of 112 were repaired and replaced. Most of the remainder 

803 were replaced gradually a few at a time utilising the spare pool in hand. 

804 J^.S. Single Event Upsets 

805 A consequence of the power supplies being located in the radiation envi- 

806 ronment of the collision hall was that both the crates and supply boards were 

807 subject to single event upsets. This typically only required the crate to be reset. 
80s Even so, every reset costed a few minutes of data acquisition time. 

809 The SY527 crates had a Lemo input for remotely resetting the crate; the 

sio reset cable ran from the crate to the counting room outside the collision hall, 

an Thus, the shift crew could reset the crate when necessary, although the pro- 

812 cedure, from indentification of a problem to manually turning the supply on 

so again, took about 10 minutes. In order to eliminate any delay due to human 

8« intervention, an automatic system was developed to detect when a particular 

sis crate has stopped communicating its voltage and current readings. Once this 

816 state was detected, the automatic reset system sent the reset signal to the crate. 

817 The supplies had to be turned on again once the crate had rebooted. Recovery 
sis from an automatic crate reset was automated with the development of Silicon 
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819 Auto- Recovery, described in Section l8~Tl which reduced the downtime from such 

820 incidents from ten minutes to less than five minutes. 

821 The supplies themselves were also subject to radiation induced effects. Specif- 

822 ically, certain power metal oxide semiconductor field effect transistor (MOS- 

823 FET) components of the LOO supplies underwent single event burnouts (SEBs) [3J 

824 causing the supply to output its maximum bias voltage. If this were to happen 

825 with the detector connected, it could result in permanent damage to the sili- 

826 con sensors. Fortunately these SEBs were first observed during commissioning, 
82? before the detector was connected. 

828 In order to prevent potential damage, compact voltage fuses, called crowbars, 

829 were developed and installed. The crowbars interrupted the current if the bias 

830 voltage exceeded a specified voltage. They were placed between the high voltage 

831 detector cable and the supply itself. The initial crowbars protected the sensors 

832 from bias voltages above 150 V. In 2008, when radiation damage necessitated 

833 operating at larger bias voltages, new crowbars with a voltage limit of 450 V 

834 were installed. 

835 4-4- Operational Experience 

836 Ten years of experience in operating the power supplies helped the detector 

837 operation crew identify potential problems and react to them before they caused 

838 significant downtime. Many improvements were made, mostly to the procedures 

839 used to test supplies after repair or work in the collision hall. 

840 4-4- 1- Testing Procedures 

841 All the repaired supplies were run through a series of tests, designed to 

842 mimic the operating conditions in the collision hall, before their installation in 

843 the detector. These included long periods of being powered on (burn-in), as 

844 well as rapid power cycling. The former were crucial to detecting intermittent 

845 problems. 

846 The testing procedure consisted of connecting the supply to a set of static 

847 impedance loads (a loadbox), and turning the supply on for approximately 24 

848 hours. It was followed by a test that turned the module on and off every 2 

849 minutes for approximately 24 hours. The currents and voltages read out in each 

850 cycle were analyzed to ensure stability. 

851 If a module was forcibly switched off for its protection ( "tripped" ) during the 

852 first 24 hours or any currents or voltages were unstable during the second, the 

853 module was sent for additional inspection or repair. The safety features of the 

854 module were also tested to verify it tripped properly under limiting conditions 

855 of voltage and current and when the supply enable signal was absent. 

sse In addition, the power supplies were tested in the collision hall just before 

857 their deployment in the detector. The procedure to check out a supply in the 

ass collision hall consisted of connecting it to a loadbox and turning it on. Currents 

859 and voltages as read back from the module were recorded, and voltages were 

860 measured and compared at the loadbox. The safety features were tested again to 

861 make sure that the supply tripped off at intended voltage and current conditions. 
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862 Then, the supply was connected to the sensors, and turned on. Given the 

863 location of the crates, the RS-232 interface provided the optimal method of 

864 testing, as it allowed easy control of the module using a laptop on the collision 

865 hall floor. 

see The loadbox was made by the Computing Division ESE at FNAL and in- 
867 corporated different load resistances for LOO and SVX-II/ISL supplies with ca- 
ses pability to switch between them. It utilized an ADC to automatically measure 

869 the voltages on each channel. With the loadbox, the collision hall check-out 

870 procedure was quick, reliable and reproducible. 

871 4-4- Damage to Power Supplies During Transportation 

872 When power supply modules could not be repaired on site, they were sent 

873 back to the manufacturer in Italy. Shipping the modules back to Fermilab 

874 damaged more than half of the shipped power supplies. In 2008, a short study 

875 was done to examine the shipping method adopted by the manufacturer and 

876 implement improvements to prevent future breakage. 

877 In order to streamline the transit and prevent breakage, a set of procedures 

878 was devised for the manufacturer to follow when shipping supplies back. First, 

879 instead of routing via CAEN's business office in New York, the modules were 

880 required to be shipped on a direct flight from Italy to Chicago. Second, the 

881 supplies were to be shipped on a standard shipping pallet, requiring the use of 

882 a forklift to prevent the box from being mishandled. Finally, an acceleromcter 

883 inside the box and shock/tilt sensors affixed to the outside of the box were used 

884 to monitor the detailed motion the supplies were subjected to while in transit, 
ass Using these procedures, every supply that was shipped back to Fermilab arrived 
886 in working order. 

as? 5. Cooling System 

ass 5.1. System Overview 

889 The sensors had to be kept chilled at all times in order to prevent migration 

890 of radiation-induced defects, which shorten the working lifetime of the detector, 

891 and to reduce the sensor leakage current, which increases with radiation damage. 

892 The SVX-II, ISL, and LOO detectors were cooled by two closed-circuit liquid 

893 cooling systems. One system, shown in Fig. [TU1 was used to cool SVX-II and 

894 LOO. A second separate system was used for ISL, although some of the cooling 

895 control electronics and interlocks were shared. 

896 Due to the higher radiation levels close to the beam line, the operating tem- 

897 perature of LOO and SVX-II was chosen to be lower than ISL. The temperature 

898 of the coolant out of the chiller was -10 °C for SVX-II/L00 and +6 °C for ISL. 

899 The coolant for SVX-II/L00 was a mixture of 30% ethylene glycol and 70% wa- 
rn ter by weight, and the coolant for ISL was distilled water. Both coolants warmed 

901 up by a degree or two in the piping between the chillers and the detector. The 

902 total cooling load for SVX-II/L00 during operation was approximately 5 kW, 
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Figure 10: Schematic of the SVX-II/L00 cooling subsystem. The subsystem for ISL can be de- 
scribed in the same manner, though both subsystems share the same PC and control/interlocks 
crates. 
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903 and that of ISL is approximately 4 kW. About half of this heat load was pro- 

904 duced by the silicon detector and the remainder was heat transferred from the 

905 warmer surroundings. Most of the heat generated by the silicon detector was 

906 produced by the SVX3D readout chips. All of the portcards were cooled by the 

907 ISL system because the temperature of the ISL system was within the range of 
90s the optimal performance of the light transmitters. In order to prevent water 

909 from condensing on the sensors and electronics, dry nitrogen flowed through the 

910 silicon detector volume. 

on In the SVX-II ladders, the electrical hybrids were cooled through thermal 

912 contact to beryllium support bulkheads, with integrated cooling channels. For 

913 LOO and ISL ladders, and for the portcards, cooling was achieved through ther- 
9« mal contact to aluminum tubes glued to the mechanical supports. The SVX-II 

915 and ISL sensors were not in close thermal contact with the coolant, however, 

916 nor were their temperatures directly monitored. Based on the measurements 

917 of the ambient temperatures and the bulkhead temperatures and on thermal 

918 models, we estimated the temperatures of the sensors while the detector was 

919 powered to be between and 10 °C for SVX-II, and between 15 and 25 °C for 

920 ISL. The LOO sensors, however, were in close thermal contact with the coolant 

921 tubes, and their temperature was about -5 °C when powered. 

922 In order to prevent damage to the electronics if a coolant pipe leaked, all 

923 cooling pipes were operated below atmospheric pressure so that a leak in a 

924 pipe or a fitting would draw nitrogen into the cooling system rather than leak 

925 coolant into the detector volume. The cooling system had vacuum pumps and 

926 air separators in order to maintain the sub-atmospheric pressure in the system at 

927 all times. Two vacuum pumps were available per system, with one running and 

928 one piped in as an immediately available spare, switchable with electronically 

929 controlled valves. 

930 The coolant was circulated by pumps on the two chillers: one for SVX- 

931 II/L00, and one for ISL. A third chiller was available to be used as a spare in 

932 case of failure of either chiller. This system was used successfully in November 

933 2004, when the spare chiller was used for two days while a leak in the ISL chiller 

934 was identified and fixed. 

935 Each of the SVX-II/L00 and ISL chiller circuits had a set of filters — a 

936 microparticlc filter, a UV sterilizer to limit biological activity in the coolant, and 

937 a resin-cartridge de-ionizer. The conductivity of the coolant was regulated to be 

938 approximately 0.6 ^iS/cm. The pH of each of the two coolants was monitored 

939 via weekly samples drawn from the air separators and both stayed near a pH of 

940 6. 

941 To ensure the safety of the silicon detector, a series of interlocks prevented 

942 the power supplies from being turned on when insufficient cooling was avail- 

943 able, and coolant flows were switched off if an unsafe situation existed. If the 

944 pressure in any of the cooling lines rose to within 1 psi of atmospheric pressure, 

945 electrically controlled solenoid valves on the coolant supply lines to the affected 

946 detector subsystem were shut automatically. If dewpoint sensors detected that 

947 the dewpoint was within 3°C of the minimum temperature in the detector 

948 volume, then flows were shut off. 
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949 The interlock system had the ability to disable power to the detector. The 

950 CAEN power supply modules required a voltage of +5 V in a Lemo connector 

951 in order to enable detector power. Dropping of this +5 V had the effect of 

952 switching the power off. If there was insufficient flow of coolant to the detector 

953 or if temperature sensors indicated that a coolant temperature was too high 

954 then the detector power supplies are turned off via the +5 V control lines. The 

955 electronics crates that housed the CAEN modules monitored the temperature 

956 of the electronics and the status of the crate fan pack, and would shut off if a 

957 failure was detected. 

958 These interlocks were controlled by a Siemens SIMATIC 575 Programmable 

959 Logic Controller (PLC) [33J, attached to two crates containing modules that 

960 read out temperature, pressure, and flow transducers. A third crate provided 

961 control for the solenoid valves, the vacuum pumps, the chillers, and the 5 V 

962 power supply enable lines. 

963 A second layer of interlocks was provided by a Siemens QUADLOG CCM+ 

964 PLC [34(. This controller had an independent readout of the coolant flows 

965 and pressures and also monitored the state of the power supply interlocks. If 

966 power supplies were permitted to be turned on but coolant flows were too low 

967 (< 1 LPM) — which could only happen if the first PLC's interlocks had failed 

968 - this interlock system would turn off the power to the silicon power supply 

969 and FIB racks (section [3]). The QUADLOG system also protected against over- 

970 pressure conditions — again in the event of a failure of the first PLC system — 

971 by shutting down coolant flows. 

972 During a power outage, backup systems kept the cooling system functioning 

973 at a reduced level. Power was supplied to the PLCs and control electronics 

974 from an uninterruptible power supply (UPS), which was backed up with a diesel 

975 generator. The vacuum pumps were powered by the diesel generator but not the 

976 UPS, and therefore they did not pump during the time required for the diesel 

977 generator to start up at the beginning of a power outage, but resumed pumping 

978 shortly into the outage. CDF's building water chillers did not operate during a 

979 power outage, and so a dedicated air-cooled backup chiller, which was powered 

980 by the diesel generator, was able to supply chilled water to the SVX-II/L00 
osi chillers. The ISL chiller's compressor did not operate during a power outage, 

982 but a backup coolant pump maintained coolant circulation. 

983 5.2. Operational Experience 

984 The cooling and interlock system for the CDF silicon detector had a high 

985 reliability and failed infrequently. The main goals of protecting the silicon detec- 

986 tor and preventing any damage to other detectors were well fulfilled. However, 

987 a few major incidents affecting the ISL cooling structure inside the detector 
gas revealed the inadequacy of the interconnected branching scheme of the cooling 

989 piping (see Fig. [11]). Isolation of leaky segments (Section [5.2.3)) was very dif- 

990 ficult and required a multi-week shutdown of the detector. During these down 

991 times a great deal of work was invested to investigate and repair leaks. In con- 

992 trast, the SVX-II/L00 cooling system, which had a simpler geometry, was more 

993 stable and performed better. 
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Figure 11: Diagram of the connections of the cooling lines of the east half of ISL showing the 
supply and return manifolds located in the detector frame. Flow which enters the detector 
via one of the four supply lines was be shared among several return lines. Electronic supply 
valves and manual return valves are shown. 

994 5.2.1. Blockage of Cooling Flows in ISL 

995 When the ISL detector was commissioned, it was discovered that 35% of 

996 the lines in ISL were not cooling. Further investigation using long borescopes 

997 showed that the coolant flow was blocked by epoxy found at aluminum right- 

998 angle elbows in these cooling tubes. In 2002 and 2003, these blockages were 

999 opened by shining Nd:YAG laser light to vaporize the epoxy. A pulsed laser 

1000 operated at an average power of 10-40 W was used, guided by an optical fiber 

1001 with a 400 /Ltm core and a 20 /jm Al jacket. The fiber was attached at the end to 

1002 a device holding a prism at a right angle so that the laser light could be aimed 

1003 at the epoxy plugs just beyond the bends of the elbows. 

1004 This operation was delicate and difficult, as the coolant piping had an inside 

1005 diameter of 4 mm and the elbows were approximately two meters away from 

1006 the accessible end of the tubing. One cooling line was found to be extremely 

1007 difficult to open and during the attempts the prism holder became detached 
loos from the fiber and remained lodged inside the pipe. The flow in the line was 
iooo not restored, and a concern of leaks developing due to stagnant coolant with 
ioio corrosive ions building up made it prudent to plug the narrow tube. Aluminum 
ion plugs were inserted at the manifolds where the cooling supply and return lines 

1012 divide into four or five narrower tubes. A second prism holder was stuck in 

1013 another narrow tube and repeated attempts to remove it failed. This reduced 

1014 the flow, but did not block it. 

1015 The successful opening of the blocked ISL flows raised the fraction of working 

1016 ISL cooling lines from 65% to 96%. 
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1017 5.2.2. Degradation of Coolant and Corrosion 

low In 2005, the SVX-II chiller setpoint was lowered from — 6°C to — 10 °C in 

low order to extend the longevity of the silicon detector. After this change to the 

1020 operating temperatures, there was an incident during routine maintenance work 

1021 on the ISL vacuum pump that accidently fired the safety interlocks of the cooling 

1022 system. This stopped the flow of ISL coolant for at least 30 minutes. The ISL 

1023 cooling system also serves as the cooling for the portcards for the entire silicon 

1024 detector and during this period, the coolant in the SVX-II portcard cooling 

1025 lines began to freeze due to their proximity to the SVX-II cooling system which 

1026 was at -10 °C. This constricted the flow of coolant in the SVX-II portcard lines 

1027 to less than the accepted minimum rate for the safety interlocks which in turn 

1028 did not permit the SVX-II detector to be powered. The frozen coolant in the 

1029 SVX-II portcard line was melted by raising the SVX-II coolant temperature from 

1030 -10 °C to +6 °C and turning the ISL on to raise the ambient temperature. After 

1031 two hours, flow was reestablished to the SVX-II portcard lines which allowed 

1032 the SVX-II detector to be powered. In order to prevent freezing incidents, which 

1033 risk portcard line ruptures, 10% of ethylene glycol by weight was added to the 

1034 ISL coolant. 

1035 In 2007 the pressure in an ISL portcard supply line rose beyond the op- 

1036 erational tolerance due to leaks in the aluminum manifold which distributes 

1037 the coolant to the portcards. Sufficient flow could not be maintained to cool 

1038 the silicon sensor readout electronics and the east half of the detector had to 

1039 be switched off. The investigation found that the pH of the ISL coolant had 

1040 dropped to approximately 2.0. The conductivity had risen from 2 /iS/cm to 

1041 around 3000 /iS/cm. Unfortunately, the conductivity meters at that time satu- 

1042 rated at values far below 3000 /iS / cm. An analysis based on ion chromatography 

1043 revealed that the ethylene glycol had degraded into light organic acids, mainly 

1044 formic acid at a concentration of 12.5 g/1 (0.265 moles/1). Another analysis 

1045 ruled out the possibility of microbial-induced degradation. There was evidence 

1046 that warming up the ISL detector to 13 °C during a two month shutdown in 

1047 the summer of 2006 accelerated the acidification in correlation with the rise in 

1048 conductivity. Several system components corroded faster than others with this 

1049 degraded coolant. Outside the detector, the brass valve stems of the solenoid 

1050 flow control valves had corroded, causing failures in the valves weeks before the 

1051 operational collapse. The portcard manifolds were made with aluminum 5052 

1052 piping welded using aluminum 5356 filler [35$ . The filler material had corroded 

1053 more quickly than the piping material. 

1054 The system was successfully repaired during a shutdown in the summer of 

1055 2007. The repair work involved inspection of the system with a borescope. The 
lose affected manifolds were located at a distance of 1 m inside the cooling tubes. 
1057 With the use of a custom-made tool, Scotch DP190 epoxy (3(| was laid down 
loss on the welded area of the leaking manifolds. In order to prevent corrosion 
1059 the coolant was replaced by deionized water, and the pH and conductivity were 
loeo monitored frequently in order to identify and mitigate hazards quickly. Through 
io6i the remainder of Run II, the pH and conductivity were stable, and the affected 
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1062 portions of ISL were cooled. 

1063 The +6°C cooling water supplied to the portcards and the heat generated 

1064 by the portcards when powered were sufficient to keep the water from freez- 

1065 ing during normal detector operation. When the portcards were not powered, 

1066 however, a freezing hazard existed. Protection against freezing (the original 

1067 impetus for adding glycol to the ISL coolant) was implemented via an inter- 
ns lock that inhibited flow to the SVX-II when the temperature measured in any 

1069 portcard coolant circuit fell below 1 °C. 

1070 5.2.3. Leaks in ISL 

1071 The overall leak-rate of the ISL coolant system increased steadily after in- 

1072 stallation. The leak rate, as determined by the amount of time it took to leak 

1073 up to atmospheric pressure when the vacuum pumps were valved off, increased 

1074 by a factor of five between 2007 and 2009, prompting a third intervention to 

1075 extend the longevity of the ISL cooling system. The aluminum ladder-cooling 

1076 tubing was found to be leak-tight, except for a few smaller leaks in two tubes. 

1077 The epoxied joints between polyethylene tubing and the aluminum manifolds 

1078 where the small aluminum cooling pipes join were found to be the leakiest in 

1079 the system. Additional epoxy was applied to three of these joints, and the flow 
loso performance improved, but the overall nitrogen leak rate remained at compara- 
losi ble levels as measured by the exhaust flow rate out of the vacuum pump. The 

1082 flow rates and cooling performance were monitored closely for the remainder of 

1083 CDF's data-taking run, and did not degrade to the point of requiring a change 

1084 in the operation of ISL. 

loss 6. Particle Beam Incidents and Monitoring 

lose The particles from standard Tevatron running conditions (physics runs with 

1087 proton-antiproton collisions) were responsible for the vast majority of the ra- 

1088 diation dose to the CDF silicon detector sensors and components. However, 

1089 beam instabilities and sudden beam losses were an unavoidable part of running 

1090 a large accelerator, and posed a threat to particle detectors. As described in 

1091 Section [21 the silicon detector was the closest to the beam and suffered larger 

1092 consequences than the other CDF sub-detectors. Beam incidents early in Run 

1093 II resulted in large and acute radiation fields that permanently damaged about 

1094 4% of the readout chips in the detector. A two-pronged approach was taken to 

1095 reduce the possibility of additional damage from beam incidents: a thorough 

1096 review of past incidents and a strict beam monitoring system. 

1097 6.1. Particle Beam Incidents 

1098 When a beam incident occurred, a thorough review of the problem, in collab- 

1099 oration with the Fermilab Accelerator Division, often resulted in stricter testing 
noo of any hardware involved in the incident, as well as procedural changes in ac- 
noi celerator operation. The following list briefly describes the main types of beam 
1102 incidents and the measures taken to lessen their impact. 
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1103 6.1.1. High Beam Losses 

1104 Particles leaving the outer halo of the beams at CDF (beam losses) were 
no5 measured by counting hits in scintillation counters located on both sides of 
Hoe the detector. The counters were gated to exclude hits coincident with proton- 
iio7 antiproton collisions at the center of the detector. High losses at any time were 
lios indicative of higher radiation fields in the detector volume. More importantly, 
nog sudden changes in the losses indicated potential instabilities in the Tevatron 
mo beam. Monitoring software, described in section I6TS1 would automatically ramp 
mi down the silicon sensor bias voltage after dangerous beam conditions were de- 

1112 tcctcd. Under exceptional circumstances when the radiation increased dramati- 

1113 cally, fast hardware protection systems would issue an abort which immediately 
in4 removed beam from the Tevatron (Section 16.21) . 

ins 6.1.2. Kicker Magnet Pre-fires 

me A set of 10 abort kicker magnets were used to remove the circulating proton 

iii7 and antiproton beams from the Tevatron by steering them into a beam dump, 

ins The abort kickers have a finite rise time, so any beam which passed through 

mo those magnets as their field ramped up were not cleanly extracted into the 

1120 beam dump. The beam train structure contained an unpopulated 1.4 fis gap, 

1121 known as the abort gap, to allow the abort kickers to ramp up without affecting 

1122 circulating beam. 

n23 Normally, beam aborts were synchronized with the abort gap so that the 

H24 kickers would reach nominal field before the first bunches arrived to be sent to 

H25 the dump. Occasionally, one of the 10 thyratrons that powered the individual 

H26 abort kickers would trigger spontaneously. When such a pre-fire was detected, 

H27 the other abort kickers were fired intentionally, without synchronizing to the 

n28 abort gap, in order to abort the beam as quickly as possible. Any beam that 

n29 passed through the pre-fired and other kicker magnets before they had reached 

mo nominal field could continue traveling with a distorted orbit, possibly hitting 

mi accelerator components and creating secondary and tertiary showers of particles 

H32 at the experiments. The location of the abort kickers relative to the detector 

H33 made CDF susceptible to large, acute doses from proton initiated showers. One 

1134 such incident in 2003 resulted in the loss of about 4% of the silicon readout 

ins chips. After a thorough review of the incident, a new collimator was installed 

H36 to intercept protons that would strike the CDF detector due to an abort kicker 

in? pre-fire. Although pre-fires continued to occur several times per year, after the 

H38 installation of the new collimator, the silicon detector did not sustain significant 

n39 damage from such incidents. 

n4o 6.1.3. Quenches 

1141 A quench is the sudden transition of a superconductor from a state with 

1142 zero electrical resistance to a normal state with small, but finite, electrical resis- 
H43 tance. For superconducting magnets like those in the Tevatron, a quench could 
H44 be caused by a temperature rise of the current-carrying superconducting cable 
H45 above its critical temperature. This could be caused by localized beam losses in 
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1146 the magnet or a loss of cryogenic cooling. An automated quench protection sys- 

1147 tern protected the magnets from potential damage caused by the sudden ohmic 
ins heating, generated by the large current powering the magnets when resistance 

1149 became normal. The quench protection monitors (QPMs) monitored the resis- 

1150 tive voltage across a string of several magnets. When a quench was detected, 

1151 the QPM simultaneously energized heaters within each magnet to enlarge the 
H52 quenched region and enabled switches to bypass current out of the affected mag- 
H53 nets. In addition, it initiated a beam abort, to reduce the impact of the orbit 
H54 distortion caused by the decaying magnetic field of the quenched magnets. 

H55 A key component of protecting the silicon detector was the Tevatron QPM 

use being able to detect a quench and abort the beam as early as possible. The 

us? early Tevatron QPM operated at 60 Hz, leading to quenches possibly remaining 

use undetected for up to 16 ms (>760 beam revolutions) between QPM measure- 

H59 ment cycles. Indeed, this shortcoming was at the heart of an incident in 2003 

nee that caused considerable damage to the Tevatron and to the experimental de- 

ii6i tcctors. When a movable experimental detector (Roman Pot) suddenly moved 

H62 toward the beam, high beam losses scattered from the device caused a very 

lies fast quench of nearby superconducting magnets that likely went undetected for 

H64 the entire gap between QPM measurement samples. The beams circulated for 

lies most of that time with highly distorted orbits that showered the experiments 

nee and accelerator components. A stainless steel collimator had a groove bored 

H67 into its surface over half of its 1.5 m length. A review of the incident led to 

nee a higher bandwidth upgrade of the QPM system (to 5 kHz) completed in 2006 

H69 which allowed quenches to be detected and beam aborted in ~ 500 ^s (25 beam 

1170 revolutions). 

1171 6.1.4- Separator Sparks 

H72 In the Tevatron, the proton and antiproton beams circulated within a single 

H73 beam pipe. Electrostatic separators kicked the beams onto distinct helical orbits 

H74 so that head-on collisions occurred only at the CDF and DO interaction points. 

H75 These 26 separators were stainless steel, parallel-plate electrodes, 2.5 m long, 

H76 with a 5 cm gap operating with a gradient of up to ^40kV/cm. Occasionally, 

H77 a high-voltage breakdown (spark) occurred between the plates, or between a 

H78 plate and the surrounding shell. The effects of a separator spark depended on 

H79 which separator broke down, and when the spark occurred during a Tevatron 

liso cycle. Such breakdowns caused a momentary kick to the beams resulting in orbit 

lisi distortions that caused beam loss spikes, emittance growth, and a small drop in 

H82 instantaneous luminosity. The orbit distortion caused by a separator spark could 

H83 be large enough to drive beam into collimators and cause quenches of nearby 

H84 superconducting magnets. Improvements in the high voltage conditioning of the 

lies separators reduced the overall spark rate compared to early Run II conditions, 

use During the latter years of Tevatron operations, only one or two stores per year 

us? terminated prematurely from a separator spark. 
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1188 6.2. Beam Monitoring and Detector Protection 

nag CDF had a slow-reacting (of the order of seconds) software monitor and a 

1190 fast-reacting (of the order of 10 fxs) hardware protection system to help prevent 

1191 damage to the silicon detector from the incidents mentioned in Section 16.11 

H92 The software program, called TevMon, collected several measurements that 

H93 described beam conditions, some of which were provided to CDF by accelerator 

H94 monitoring systems. These included beam losses, RF station voltages, instan- 

H95 taneous luminosity, abort-gap beam current, and abort kicker magnet voltages. 

H96 A variable that entered the warning range caused an audible alert for the shift 

H97 crew, which indicated degraded beam conditions that may warrant their atten- 

H98 tion. When TevMon reached the alarm state, the bias voltages of the silicon 

H99 detector were turned off automatically. 

1200 The fast reacting hardware protection systems, the beam condition moni- 

1201 toring (BCM), consisted of four Beam Loss Monitors (BLMs). These were two 

1202 ionization chambers on the east and two on the west side of the detector, about 

1203 4.3 m from the nominal interaction point (LP.) and at a radius of about 20 cm 

1204 from the beam axis. The location of the BLMs from the LP. was necessary due 

1205 to their size, which made it impossible to put them inside the detector. The 

1206 BLMs were read out every 210 /is (10 beam revolutions), and a circular buffer of 

1207 2048 measurements was kept. The BLMs monitored the radiation accumulated 

1208 over the last minute and the radiation rate. An accumulated dose greater than 

1209 19 rads (0.19 Gy) over the previous minute would issue an integrated dose alarm, 

1210 whereupon the shift crew would pass on the alarm to the Tevatron operators 

1211 and adjustments to the Tevatron operation would be made. If the radiation rate 

1212 exceeded 12 rads/s (0.12 Gy/sec), the radiation abort alarm would fire, where- 

1213 upon the Tevatron would automatically issue an abort and the beams would be 

1214 dumped. 

1215 A closer examination of beam incidents showed the BLMs lacked the timing 

1216 resolution and dynamic range to foresee the conditions leading to an radiation 

1217 abort. A system of smaller sensors, closer to the beam and with a faster read- 
me out system, could abort the beam more rapidly to improve the safety of the 

1219 CDF silicon detector. This led to the installation of a diamond-based BCM 

1220 system [33] ■ A total of thirteen diamond sensors were installed in the CDF de- 

1221 tector, at the locations indicated in Fig. [12] Two groups of four diamond-based 

1222 sensors were located inside the tracking volume, with each group mounted in a 

1223 support structures at a distance of 1.7 m from the nominal LP., arranged as the 

1224 sides of a 4 cm by 4 cm square. Five more diamond-based sensors were installed 

1225 outside the tracking volume, on the previous BLM system support structure; 

1226 two diamonds on the west side and three on the east side. 

1227 The BCM system was configured to abort the Tevatron beams when at 

1228 least four diamond sensors measured a current of at least 500 nA and the CDF 

1229 solenoid was fully energized, as the diamond dark current was affected by the 

1230 external magnetic field. These settings were determined from a six-month study 

1231 period to optimize the thresholds, and allowed quick response to potential beam 

1232 incidents, while minimizing the number of false aborts. 
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Figure 12: The locations of the diamond- based BCM system in the CDF detector. The 
picture shows the upper hemisphere of the CDF detector. The proton beam circulated from 
left-to-right along the horizontal axis, colliding with the anti-proton beam that circulated in 
the opposite direction. The collision occurred at the interaction point, indicated by "LP." in 
the figure. A pair of four red dots, symmetrically placed on either side of the LP., indicate the 
sensors inside the tracking volume, and the red dots further away from LP. along the beamline 
indicate sensors outside the tracking volume. 

1233 7. Sensor Readout Calibration 

1234 The dynamic pedestal subtraction (DPS) feature of the SVX3D chip used 

1235 a fraction of all channels to calculate a real time, common pedestal set to cor- 

1236 respond to zero collected charge. This algorithm ensured that most channels 

1237 have a mean pedestal close to zero, but some channels still exhibited a pedestal 

1238 significantly different than the DPS pedestal. 

1239 The calibration algorithms measured the mean pedestal and noise for each 

1240 electronics channel on every chip. These values were determined from dedicated 

1241 calibration runs, in which every channel was read out, taken with colliding 

1242 beams at low (< 50 x f0 30 cm _2 s _1 ) instantaneous luminosity. A standard 

1243 silicon calibration included two runs: one with DPS turned off and one with 

1244 DPS on. The analysis of data from these runs started with accumulation of 

1245 the ADC pulse height distributions for every strip in the silicon system. The 

1246 calibration algorithms used the DPS-off pulse height distribution to infer the 

1247 DPS-on distribution. An example of the pulse height distributions from one 

1248 silicon strip in DPS-off and DPS-on calibration runs is shown in Fig. If 31 

1249 The DPS-on pulse height distributions alone are not sufficient for estima- 

1250 tion of the DPS-on pedestals and noise. To deduce these pedestals, the software 

1251 needed to simulate the effect of the DPS circuit using DPS-off data. The soft- 

1252 ware required the number of channels above threshold before the integration 

1253 counter begins. This value was determined for each chip during commissioning 

1254 and an average value of 33 channels was found adequate for data taking. 

1255 After the pedestal and noise (width) of the ADC distributions were found for 

1256 each channel using robust estimators, a program found channels to be flagged as 

1257 bad, due to low occupancy ("dead") or high occupancy (noisy or "hot"). Noisy 
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Figure 13: Effect of Dynamic Pedestal Subtraction on pulse height distribution of a single 
channel in 1000 events. The DPS-off distribution is shown in blue, DPS-on is red, and the 
green histogram is an extrapolation of what the DPS-on distribution would be if negative 
ADC counts could be recorded. The peak at ADC counts in the DPS-on distribution is 
suppressed by choice of scale. 
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1258 channels were not included in the default offline clustering algorithm, but could 

1259 be included in special studies. 

1260 Once bad channels were tagged and calibration constants calculated sepa- 

1261 rately for DPS-on and DPS-off operation, a series of calibration quality checks 

1262 were performed. Calibration constants that changed dramatically were flagged 

1263 for expert notification. Since the offline calibration runs were performed ap- 

1264 proximately twice per month, the configuration of ladders could change between 

1265 calibrations. In these cases, the calibration framework mixed the DPS-on and 

1266 DPS-off pedestals and noise constants as appropriate and wrote the relevant 

1267 constants to the CDF database. The calibration constants were used for data 

1268 reconstruction as well as detector simulation. 

1269 8. Detector Monitoring and Operations Support 

1270 The operation and maintenance of the silicon detectors required dedicated 

1271 personnel and software for fast problem response. The CDF Silicon Group was 

1272 comprised of approximately ten on-call experts who provided 24-hour support 

1273 to the CDF operations group and performed regular maintenance to ensure 

1274 optimal performance of the detector. The group was led by two sub-project 

1275 leaders. Two experts were assigned to each of the following subsystems: DAQ, 

1276 power supplies, and cooling. Another expert periodically performed calibrations 

1277 of the detectors and ensured their quality. In addition, three to four experts 

1278 provided online monitoring of the detector and gave prompt feedback to the 

1279 group of any developing hardware problems. The group strived to optimize and 

1280 automate most aspects of the silicon detector operations over the years. 

1281 Many software packages were developed for detector operations and moni- 

1282 toring. They were broadly divided into three categories: stand-alone java appli- 

1283 cations used by the shift crew, scheduled jobs to acquire data on power supplies 

1284 and the cooling and DAQ systems, and Perl-based CGI scripts that provided 

1285 real-time information to the Silicon Group. 

1286 8.1. Silicon Auto-Recovery 

1287 The SVX3D readout chips (see Section [3 .3. 1[) required proper initialization 

1288 sequence as soon as they were powered. The process of turning on power to 

1289 the silicon detectors required synchronized actions in both the power supply 

1290 control and the DAQ. Over time, a set of well established procedures to recover 

1291 from common failure modes was developed. However these manual recovery 

1292 procedures reduced the data-taking efficiency, especially when a large fraction 

1293 of the detector channels needed to be turned on after failures, such as a power 

1294 supply crate reset (see Section |4~3|) . 

1295 In order to automate and speed up the recovery of power to the detector 

1296 channels, a Silicon Auto-Recovery (SAR) software tool was developed. SAR 

1297 detected channels that lost power during data-taking and sent a request to the 

1298 power supply control to turn these channels back on. Once the power to the 

1299 chips was restored, SAR took the corresponding FIB through the initialization 
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1300 process which also initialized the chips. During this process, data taking was 

1301 suspended. After automating both the power supply crate reset and recovery, 

1302 the average experimental downtime due to lost communication (see Section [4.3p 

1303 with the power supply crates was reduced from 10 minutes down to less than 5 

1304 minutes. 

1305 8.2. SVXMon 

1306 SVXMon was the monitoring application used for both online and offline 

1307 diagnostics of the CDF silicon detector problems. It ran continuously during 

1308 data taking as part of a set of CDF monitoring applications. It accumulated 
woo various statistics and presented a coherent set of silicon performance plots to 
mo the shift crew. On special occasions, it sent automated requests to reinitialize 
mi DAQ components showing problems. 

1312 SVXMon was a highly configurable program capable of presenting both very 

1313 general and very detailed views of the silicon data. For each silicon strip, SVX- 

1314 Mon accumulated the number of hits and pulse height distribution. These were 

1315 used to create plots of occupancies, average pulse heights, distribution shapes, 
me etc., with various degrees of detector granularity. The monitoring application 

1317 had a large number of configuration parameters which evolved over time to op- 

1318 timize the information useful to detect and understand error conditions in the 
mo silicon detectors. 

1320 8.3. IMON 

1321 IMON was an application used to monitor currents in the silicon detectors. 

1322 IMON displayed each ladder of the detector as a set of color-coded boxes (one 

1323 for each channel of the ladder). If the measured current was within a pre-set 

1324 range, then the box showed up green. If it was just outside the good range, 

1325 it showed up yellow. If it was far outside that range, it turned pinljf] When a 

1326 ladder turned pink, it alerted the shift crew so they could take action. 

1327 As the sensors degraded due to exposure to radiation, they drew more cur- 

1328 rent. Eventually, a sensor drew enough current to send it over the pre-defined 

1329 "good" range; this was normal and simply required adjustment of the good 

1330 range for that particular sensor by a member of the Silicon Group. A typical 

1331 bias channel drew an additional 30 fiA of current for every 500 pb _1 of delivered 

1332 integrated luminosity. The currents drawn by the chip chains did not change. 

1333 When one exceeded the normal limits, it typically needed to be reinitialized. 

1334 Although the concept for adjusting the bias current limits was simple, the 

1335 actual procedure was tedious. Limits for the bias currents were adjusted and 

1336 documented in a database. The rate of increase varied widely across the channels 

1337 due to the sensor type, radiation exposure, and other causes. With close to 500 

1338 bias channels, 25-50 adjustments were made every week. Changing the limits 

1339 by hand could take several minutes per channel. For bias channels which had 



4 Other colors were used to indicate ladders which had tripped, turned off, had lost com- 
munication or were ignored by the DAQ. 
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1340 increases consistent with normal aging, the limits were adjusted by a monitoring 

1341 program. After automatic adjustment software was made operational, it took 

1342 care of approximately 95% of the limit adjustments, drastically cutting down 

1343 the workload of the Silicon Group. 



1344 8.4. ADCMon 

1345 The Silicon Group had the responsibility to ensure good quality of the data 

1346 collected by the silicon detectors. ADCMon was an application developed to 

1347 ease that task. ADCMon read the raw information recorded by the DAQ and 

1348 provided the distributions of charge in ADC counts for each silicon ladder. Two 

1349 different versions of ADCMon were implemented: online and offline. The online 

1350 version operated in the CDF control room during data taking and provided data 

1351 to SVXMon. It provided the charge distribution of the last 500 events, as well 

1352 as the statistics accumulated during the entire run of data taking. The offline 

1353 version was executed with a delay of less than one day from the end of the 

1354 run and was useful to understand long term behavior of the silicon ladders. It 

1355 generated a table of histograms representing the charge distribution for events 

1356 of a given run in comparison with a reference run. It also provided information 

1357 about the percentage of bit errors in digital transmission of data. 

use Optical data transmission bits could permanently get stuck in a high or low 

1359 state due to radiation damage, cable or electronics malfunction, as described in 

1360 Section 13.8.21 This could lead to a lower resolution of the charge distribution. 

1361 Discrepancies in the shape of charge distributions or absence of data could be 

1362 caused by a FIB or power supply failure. Severely underdepleted ladders could 

1363 show an observable drop in the high end of the charge distributions. A visual 

1364 evaluation of the offline results required 5-10 minutes per day. 



8.5. iFIX 

Some components of the silicon detector needed to be monitored independent 
of data taking. A system to do this was developed based on the commercial 
automation software Proficy HMI/SCADA iFIX, licensed by GE Fanuc 



Data was stored at intervals on the order of a few seconds, with recent readings 
ranging from a few hours to a few days being displayed in the CDF control 
room. Older data was migrated to permanent storage. This information could 
be retrieved from iFIX computer nodes. 



1373 8.5.1. Slow Control of Cooling System 

1374 An iFIX node was connected to the PLC system through a VME module 

1375 to a Siemens SIMATIC 505 Crate, where the I/O modules resided (see Fig. [TT] 

1376 in Section 15.21) . Information from the cooling related devices was available in 

1377 the iFIX software which provided displays of real-time readings of the devices 

1378 (temperature sensors, pressure gauges, flowmeters, valves, etc). Changes to 

1379 parameters of selected devices could be made through this system. The system- 

1380 wide and sub-component interlock status were also displayed. In particular, the 

1381 most valuable quantities monitored in this system were the temperatures of the 

1382 cooling lines inside the detector, as well as the flows and pressures. 
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1383 8.5.2. Audible Alarm System 

1384 Alarm conditions were denned for selected variables, such as the cooling vari- 

1385 ables which could potentially trigger the interlocks. In the case of temperatures, 
use pressures and flows, low and high warning and alarm limits were defined. Other 

1387 variables in the alarm list included the high voltage and trip status of power 

1388 supplies. Any alarm going off was followed by an audible voice alarm. 

1389 8.5.3. Monitoring of Rack Modules 

1390 The status of the AC power to the racks hosting the FIB and CAEN crates in 

1391 the collision hall was monitored, and remote power cycling of racks was possible. 

1392 In addition, monitoring and alarming on low-voltage status was available for 

1393 small devices in the racks, such as fans inside the crates. This was also true for 

1394 the racks hosting the VRB modules outside the collision hall. We also monitored 

1395 the power supply output voltages for the VRB and FIB crates. 

1396 8.5.4- Reset of Silicon Power Supplies 

1397 The automatic reset of CAEN power supply crates (explained in Section |4~3")) 

1398 was monitored in the iFIX system. Additionally, the silicon operations team 

1399 could manually reset any power supply crate from this system. 

woo 9. Monitoring Radiation Damage Effects 

1401 9.1. Depletion Voltages 

1402 Periodically, the depletion voltage of each sensor was measured to monitor 

1403 the radiation effects on the sensor, and the operating voltage increased to ensure 
H04 that the sensors remained fully depleted. The operational definition of depletion 
H05 voltage for the CDF silicon detectors was the bias voltage at which the charge 
woe collection saturates. Specifically, the voltage at which the charge collected was 
«07 95% of the maximum value. Operating at bias voltages larger than a depletion 
1408 voltage defined this way ensured the best detector performance. 

«09 Two different methods to measure the depletion voltage were used: the noise 

1410 scan was used for double-sided sensors that had not undergone type inversion, 

1411 and the signal scan was used for all sensors. In each case, the bias voltage was 

1412 changed and either the signal from charged tracks or the noise in the sensor was 

1413 measured. The results of both methods are presented here in the context of 

1414 operating the detector. Further discussion of the observed effects of radiation 
his damage in the sensors is reserved for a future article dedicated to this topic. 

Hie 9.1.1. Noise Scans 

«i7 The noise scan measured the average noise for each ladder as a function of 

wis bias voltage. For each bias voltage setting, data were taken in read-all mode 

«i9 and the ADC count distribution for each strip was recorded. For each ladder, 

«2o the noise was determined by taking the RMS of the ADC distribution for each 

1421 strip and then averaged over the strips on the p-side (r-cj)) and the n-side (r-z) 

1422 separately. 
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Figure 14: Noise scan measurements for a single ladder from layer 1 of SVX-II at four 
different integrated luminosities (given in fb - 1 ). The blue squares are the measured noise on 
the n-side. The pale green triangles are the measured noise on the p-side of the sensor. The 
extracted depletion voltage is indicated with a vertical line. 
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The depletion voltage for each ladder was determined by fitting the n-side 
noise as a function of the bias voltage to a sigmoid function 



noise = A 



D 



1 + exp [-C(V - D)] 



(1) 



1425 where A, B, C and D are fit parameters and the variable V is the bias voltage. 

1426 The depletion voltage was identified as the voltage at which the function is equal 
to the sum of lower plateau of the sigmoid function and 5% of the height of the 
fitted sigmoid function, or Vd ep = D(lnl9)/C. 

Noise scans for a typical ladder taken at different integrated luminosities are 
shown in Figure 1141 In the early scans, the separation between the two noise 
levels for the n-side was large, and the depletion voltage was easily determined, 
as seen from Figure [T4T a). As the sensor became irradiated, the underdepleted 
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Figure 15: The average depletion voltage determined from noise scans is plotted for the 
Hamamatsu sensors in SVX-II (see Table [TJ as a function of radiation dose. 



1433 noise level decreased while the depleted noise level increased, and it became in- 

1434 creasingly difficult to determine the depletion voltage using this method. Specif- 

1435 ically, noise scans where the two noise levels of the n-side were separated by less 
W36 than 0.2 ADC counts did not give a reliable determination of the depletion volt- 
1437 age. Figure fT4Td) shows the measured noise after inversion of the sensor, for 
W38 which the p-side noise and n-side noise have similar behavior. After the inver- 

1439 sion, the overall noise level increased with radiation dose as expected, but the 

1440 shape of the curves remained the same. 

1441 The noise scan method was used to monitor the depletion voltage before 

1442 the inversion of the sensors. Figure [15] shows the average depletion voltage for 

1443 ladders in layers 0,1, and 3. Noise scans where the two noise levels of the n-side 

1444 were separated by less than 0.2 ADC counts are not included in the plot. In 

1445 order to compare the different layers, integrated luminosity was converted to the 

1446 equivalent dose of the radiation held measured inside the CDF detector with 

1447 over 1000 thermal luminescent dosimeters in 2001 39], summarized in table [2j 
K48 The behavior of the three different layers was remarkably consistent considering 

1449 that the integrated dose received by the layer depends on the distance from the 

1450 interaction region. The sensors from layers 2 and 4, which are from a different 

1451 manufacturer than the other layers, were not included in this analysis because 

1452 they developed complicated noise profiles and the simple data analysis described 

1453 above did not give reasonable quantitative results and signal scans were used 

1454 instead to monitor the depletion voltage. 

1455 9.2. Signal Scans 

1456 The signal scan provided the best evaluation of the depletion voltage, and 

1457 in many cases the only one. The charge collected by the sensor increased with 
W58 increasing bias voltage as the depleted region in the sensor grew, until the sensor 
1459 was fully depleted and the charge saturated. The scan had to be performed with 
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r (cm) dose/L (kRad/fb^ 1 ) 



LOO narrow 


1.35 


994 ± 199 


LOO wide 


1.62 


756 ± 151 


SVX-LO 


2.54 


385 ± 77 


SVX-L1 


4.12 


186 ±37 


SVX-L2 


6.52 


94 ± 19 


SVX-L3 


8.22 


66 ± 13 


SVX-L4 


10.10 


49 ± 10 



Table 2: The radiation dose per unit luminosity measured by the TLDs in the CDF tracking 
volume [39| , extrapolated to the location of the individual silicon layers . 

1460 colliding beams and required approximately two hours per layer. To minimize 

1461 the amount of lost physics data, the scans were done when the instantaneous 

1462 luminosity was low. 

K63 Data were acquired with a specific trigger selecting collision events containing 

W64 at least two tracks. From these events, tracks traversing the silicon layer under 

K65 study were identified using the COT and remaining silicon layers. If one and only 

wee one cluster existed within 150 /im of the location where the extrapolated track 

1467 crossed a sensor, the total charge of that cluster was recorded in a histogram, 

nee A reasonable fit result required at least 1,000 tracks per ladder per bias voltage 

H69 setting, with additional tracks per point needed below 20 V. 

1470 The distribution of cluster charges was fit to the convolution of a Landau 

1471 function and a Gaussian function in the region around the peak. The upper 

1472 plots of Figure[l6]are examples of this distribution at two different bias voltages. 

1473 The most probable value of the fitted function was plotted as a function of bias 

1474 voltage, and these points were fitted to a sigmoid function of the same form as 

1475 Equation [1] The measured depletion voltage was the bias voltage at which the 
we function value is 95% of the total charge is collected, or Vd ep = 0.95 * (A + B). 

1477 An example is shown in Figure [THf c). Also shown on this plot is the efficiency, 

1478 defined as the fraction of tracks for which a cluster is found. Due to the limited 

1479 data samples of these special runs, the track selection was quite loose, and 

1480 the absolute value of this efficiency does not reflect normal sensor performance 

1481 during data taking. 

1482 The non-zero value of the cluster charge for bias voltages below 10 V was a 

1483 measure of the effective clustering threshold for that ladder. All strips of LOO 

1484 were read out every event, while for SVX II and ISL, only strips above 9 ADC 
was counts plus the neighboring strip on either side were readout. The standard 

1486 offline clustering threshold was calculated from the strip noise measured in the 

1487 sensor readout calibrations (Section [7]) for the strips taking part in the cluster. 
i486 This combined with the offline clustering thresholds sculpted the cluster charge 
1489 distributions at low bias voltages. Although the initial rise of the cluster charge 
H90 was hidden by the clustering threshold, the increasing efficiency indicated in- 

1491 creasing mean cluster charge as more and more clusters were found above the 

1492 threshold. Because the strip noise increased with radiation dose, these cluster- 
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1493 ing thresholds creeped upward with integrated luminosity, and the signal scans 

1494 allowed them to be monitored. Figure [TBT d) shows the evolution with lumi- 

1495 nosity of this sensor's depletion voltage. The points beyond the inversion point 
H96 were fit with a line. 

1497 Close monitoring of LOO and the inner layers of SVX was essential after 

1498 inversion to keep the operating bias voltages above the depletion voltage. Oper- 

1499 ating voltages were increased on a sensor by sensor basis after extrapolating the 

1500 linear trend in the measured depletion voltages several months into the future. 

1501 Figure [T7] shows the linear fitss for the individual LOO ladders as gold or 

1502 red lines and the average over all the ladders as a black line and blue points. 

1503 The predicted depletion voltages for all LOO ladders lie well below the power 

1504 supply limit of 500 V and the sensor breakdown region that starts at 650 V, 

1505 and they were fully depleted through the end of Run II. Figure [18] shows the 
woe linear fits for the r-<f> side (p-side) of the first layer of SVX-II (SVX-L0). The 

1507 fitss for individual ladders are shown as red lines and their average as a black 

1508 line. The blue points are the average measured depletion voltage for all ladders. 

1509 The power supply limit for these sensors is 250 V and sensor breakdown was 
lsio expected in the range 170-270 V, indicated with a shaded region. 

1511 In agreement with the projections, roughly one third of the SVX-L0 ladders 

1512 were not fully depleted for an operating voltage of 165 V at the end of Run II. 

1513 The performance of these underdepleted ladders was only slightly compromised: 
i5« the charge collected on the p-side was reduced, while the charge collected on the 
1515 n-side was unaffected. Because of the risk of damage to the sensor, we decided 
isle not to operate any ladders above 165 V until the hit efficiency of the ladder 
1517 began to decrease. Only one ladder reached this condition by the end of the 
wis run. That is, the hit efficiency of all but one of the underdepleted ladders was 

1519 still maximal at the end of the run despite the reduced charge collection on the 

1520 r-<fr side (p-side). 

1521 9. 3. Surprises in the Behavior of Irradiated Sensors 

1522 In the traditional model for the electric field in a reverse-biased silicon sensor, 

1523 the field increases linearly through the bulk when the bias voltage is applied. 

1524 There is however evidence that trapped charge in heavily irradiated sensors 

1525 dramatically affects this simple picture of the field inside the sensor. In CDF, 

1526 we have observed evidence that, after irradiation, the field was non- uniform and 

1527 extended from both sides of the sensor. This resulted in a much longer lifetime 

1528 for SVX-L0 operating with safe bias voltages than had been anticipated. 

1529 In undamaged CDF sensors, the electric field was highest at the p-side of 

1530 a reverse biased pn diode junction, and decreased linearly through the bulk 

1531 material. At bias voltages less than the depletion voltage, the electric field at 

1532 the n-side was zero and essentially no signal was induced in the n strips as 

1533 particles passed through the sensor. The general understanding of radiation 

1534 damage to silicon sensors when the CDF detector was built (summarized in the 

1535 Hamburg model^^) was that radiation induced crystal damage made the bulk 

1536 material increasingly more p-type. At sufficiently high dose, the n-type bulk 

1537 material was expected to become effectively p-type (this is referred to as type 
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Bias voltage (V) integrated luminosity (fb' 1 ) 

Figure 16: The upper plots show the measured cluster charge distribution for a single LOO 
ladder at a bias voltage of 30 V (a) and 130 V (b) after 4 fb -1 of integrated luminosity. Plot (c) 
shows the peak of the cluster charge distribution (red circles) and the efficiency (blue squares) 
as a function of bias voltage. The dashed line indicates the depletion voltage extracted from 
the sigmoid fit. Plot (d) shows the measured depletion voltage for this ladder as a function 
of integrated luminosity, and the linear fit used to extrapolate to higher luminosity values. 
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Figure 17: Summary of depletion voltage measurements and fits for LOO wide ladders (left) 
and narrow ladders (right). 



inversion) and the junction side of the detector was expected to move from the 
p-side to the n-side. After type inversion, the electric field in the sensor was 
expected to be highest at the n-side diode junction and to decrease linearly 
through the bulk material. It was expected that essentially no signal would be 
recorded on the p-side at bias voltages less than the depletion voltage. 

This behavior is now understood to be a consequence of the properties of 
damaged silicon with an applied bias voltage. In heavily irradiated silicon, 
the trapping of leakage current charge carriers dramatically affects the electric 
field inside the sensor. Leakage current is generated approximately uniformly 
throughout the thickness of the sensor. Electrons carry charge towards the n- 
side and holes carry charge towards the p-side. This means that the density 
of moving electrons is highest near the n-side and the density of moving holes 
is highest near the p-side. Because the equilibrium number of trapped charges 
depends on the density of moving charges as well as trapping probabilities and 
trap lifetimes, the density of trapped electrons is highest near the n-side of the 
sensor and the density of trapped holes is highest near the p-side of the sensor. 
These trapped charges create an electric field with maxima at both sides of 
the sensor. The importance of trapped charges to the static field in heavily 
irradiated sensors was pointed out by Eremin, Verbitskaya, and Li in 2002 4lj. 
Swartz, et al. have tuned a two trap model to fit CMS pixel beam test results, 
including temperature dependence [42j. 

The data from the signal scans taken after significant radiation exposure are 
consistent with an electric field peaking at both faces of the sensor, and clearly 
inconsistent with the naive expectation of a linearly decreasing field with a 
single maximum at the n-side. Figure [15] shows the signal scan data delivered 
luminosities of 0.3 fb _1 (above) and 6.9 fb _1 (below) for a typical sensor in SVX- 
L0. Because the signal was induced primarily by the motion of charge carriers in 
the depleted region adjacent to the electrode, the measured cluster charge for a 
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Figure 18: Summary of depletion voltage measurements and fits for the r-r/> side or p-side 
(left) and the r-z side or n-side (right) of SVX-LO sensors. 



wee particular voltage was a measure of the size of a possible depleted region adjacent 

we? to the electrode. Because of the readout thresholds for the SVX sensors, clusters 

1568 below 10-15 ADC counts are not detected. However, the fraction of tracks with 

1569 clusters above this threshold, shown as blue squares in Figure 1191 increased as 

1570 the average charge collected increased. The upper measurement in FigurefT9lwas 

1571 done when the sensor was only slightly irradiated. The charge collection began 

1572 at smaller voltages on the p-side than the n-side, compatible with a depleted 

1573 region that began at the pn junction and grew towards the n-side electrode as 

1574 the bias voltage increases. The lower measurement was done after 6.9 fb -1 of 

1575 luminosity, and the charge collection began at similar bias voltages for each 

1576 side and increased similarly with increasing voltage. This latter behavior was 

1577 compatible with an electric field that had two maximums, one at either face of 

1578 the sensor, creating two depleted regions that started at either face and grew 

1579 toward the center of the sensor as the bias voltage increased. Similarly, for the 

1580 p-side of a single sided LOO sensor after 4 fb^ 1 of luminosity (post- inversion) , 

1581 Figure I16f c) shows an early onset and gradual increase of charge collection, 

1582 again consistent with a doubly peaked electric field. 

1583 The unexpected electric field behavior had an important consequence for the 

1584 longevity of the CDF silicon detector. The SVX-LO ladders could be operated 

1585 in a slightly underdepleted state with only a small loss in charge collection and 

1586 no loss in hit efficiency for a short time after the depletion voltage exceeded 

1587 the maximum safe operating voltage. Because the 6-tagging efficiency (Sec- 

1588 tion ll0.2|) of the detector was insensitive to a small loss in charge collection in a 

1589 fraction of the SVX-LO sensors, it was decided to operate these sensors slightly 

1590 underdepleted instead of risking damage at higher bias voltages. 
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Figure 19: The cluster charge (circles) and efficiency (squares) as a function of bias voltage 
for the p-side (r-<f>, left) and n-side (r-z, right) of a typical SVX-L0 sensor. The dashed line 
indicates the depletion voltage extracted from the measurement. The upper plots contain 
data taken after 0.3 fb _1 of delivered luminosity, the lower plots data taken after 6.9 fb _1 for 
a sensor that inverted around 1.5 fb _1 . 
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Figure 20: Measured signal- to- noise ratio for LOO (left), the r-<j> side of SVX-II (left) and the 
r-z side of SVX-II (right). 



1591 9.4- Signal-to-Noise Ratio 

1592 During Run II, the signal-to-noise ratio (S/N) of LOO and SVX-II sensors 

1593 were monitored using well-measured tracks from events selected by the low 

1594 momentum dimuon trigger. The signal S was defined as the summed charge of a 

1595 cluster of strips associated with a track and corrected for path length. The noise 

1596 for individual strips was measured during special calibration runs performed bi- 

1597 weekly with beam, as described in Section [7j The noise of a cluster N was 

1598 defined as the average noise of the individual strips belonging to the cluster. 

1599 Figure |2"01 shows the average measured S/N ratio for LOO and SVX-II, sep- 

1600 arately for the r-(j> and r-z sides. All ladders that operated consistently well 

1601 throughout Run II are included, corresponding to roughly 75% of all ladders. 

1602 As expected, the S/N ratio decreased more quickly for LOO and SVX-LO since 

1603 they were closer to the interaction point and suffered from more radiation dam- 

1604 age. The dip in S/N values for LOO and SVX-LO near 8fb _1 corresponded to a 

1605 period of slight underdepletion. 

1606 We observed plateaus in the ratios in the LOO and SVX-LO S/N ratios be- 

1607 ginning at 5 fb -1 and 8 fb -1 , respectively. To verify that the leveling-off was 

1608 not an artifact of the averaging, the S/N curves for the individual ladders were 

1609 investigated. Figure [21] shows the S/N trends for the overall LOO average and 

1610 for three typical ladders whose S/N values are close to (a) the overall LOO av- 

1611 erage, (b) the plus-one RMS variation with respect to the overall average, and 

1612 (c) the minus-one RMS variation with respect to the overall average, where the 

1613 RMS was defined as the spread of S/N values for the ladders used in the aver- 

1614 age. As we observed a plateau-like feature for each ladder, we concluded that it 

1615 was not due to an averaging artifact. The source of the plateau remains under 

1616 investigation. 
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Figure 21: Measured signal-to- noise ratio for the LOO average, and for three ladders that 
represent typical ladders close to the average, and close to the ±1 RMS signal-to-noise values 
with respect to the average. The RMS represents the spread of signal-to-noise values for the 
various ladders included in the average. 



i6i7 10. Physics Performance of the Silicon Detector 

wis Good performance of the silicon detectors was vital to the success of CDF's 

lew physics program. In this section we present some of the performance quanti- 

1620 ties which directly impact the results from analyses requiring silicon tracks or 

1621 displaced secondary vertices found by SVT. Reported here are studies on the 

1622 impact parameter resolution with and without LOO, the b-tagging efficiency and 

1623 the SVT efficiency. 

1624 10.1. Impact Parameter Resolution 

1625 The impact parameter, do is defined as the shortest distance in the r-<f> plane 

1626 between the beam line and the trajectory of the particle obtained from the track 

1627 fit. The impact parameter resolution ad is a key performance indicator of the 

1628 CDF silicon detector. This resolution affects identification of long-lived hadrons 

1629 as well as the ability to study time-dependent phenomena, such as the mixing of 

1630 B s [3~l| and charm mesons [43j]. The detector provided good impact parameter 

1631 resolution. LOO improved the performance particularly for particles with low 

1632 momentum or that passed through large amount of passive material [44| . 

1633 The resolution is parameterized as a function of the particle transverse mo 

1634 mentum px as 

= \/^ + (B/PT) 2 +rl c ^ (2) 

1635 where A is the asymptotic resolution parameter, and B is the multiple-scattering 

1636 component. The finite beam size Tbeam = 32 accounts for the uncertainty 

1637 in the location of the primary interaction. 

1638 Figure [22] shows the fitted widths of the cores of the impact parameter dis- 

1639 tributions for tracks with and without LOO hits as a function of the track pt. 
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Table 3: Fit parameters for the resolutions shown in Fig. 1221 The definitions of the fit 
parameters are given in Equation [2] A fixed beam size, rb ca m = 32 fim, is assumed. 



Track Category 


A (/jm) 


B (/um) 


Hybrid, No LOO 
Hybrid, LOO 
No Hybrid, No LOO 
No Hybrid, LOO 


35.6 ±0.6 
25.1 ±0.3 
28.6 ±0.3 
17.8 ±0.2 


77.9 ±0.5 
40.1 ±0.3 
35.5 ±0.3 
28.4 ±0.1 



The inclusion of LOO enhances the impact parameter resolution at low momen- 
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Figure 22: Impact parameter resolutions for tracks as a function of track pt- Both plots show 
the resolutions for tracks before and after the addition of LOO hits. The plot on the left shows 
the resolutions for tracks that pass through readout hybrids which are mounted on some of the 
sensors, and the plot on the right shows the performance for tracks that do not pass through 
hybrids. These plots include the »*^ oam term in Equation [2] 

1640 

1641 turn. For tracks that pass through the electrical readout hybrids, the impact 

1642 parameter resolution is somewhat degraded. 

1643 Table [3] lists the fit parameters for the four classes of tracks in Fig. [2H 

1644 The addition of LOO hits to tracks does not have much effect on the asymptotic 

1645 resolution at high momentum. Furthermore, knowledge of the impact parameter 

1646 is limited by the beam size unless a primary interacting vertex is reconstructed 

1647 with many high-momentum tracks. 



1648 10.2. b-Tagging Efficiency 

1649 Many of the physics goals of the CDF experiment rely on the identification 

1650 of weakly decaying bottom hadrons. The mean lifetime of these hadrons is 

1651 approximately 1.5 ps, and the mean decay length is order of a few millimeters. 

1652 The fact that the weakly decaying hadrons have large boosts means that the 

1653 particles from the decay travel in approximately the same direction as the parent 

1654 hadron, with their kinematic distributions depending on the mass of the parent 
less hadron. Precise measurement of the track positions allows tracks originating 

1656 from displaced vertices to be distinguished from tracks that originate at the 

1657 primary vertex. Most jets of hadrons produced in pp collisions do not contain 
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bottom or charm hadrons, and very strong rejection of falsely tagged light-flavor 
jets is another important figure of merit for the tracker. 



CDF uses a displaced- vertex algorithm, SECVTX 45], to identify — or b-tag 
— secondary vertices that are significantly displaced from the beamline. It has 
two configurations, referred to as "loose" and "tight" , which refer to the track 
and vertex requirements used to form the displaced- vertex candidates. With 
the loose requirements, more displaced vertices from heavy hadron decay are 
1665 identified than the tight requirements, but with a higher rate of falsely tagged 
light-flavor jets. 

The assignment of silicon hits to tracks has a large impact on the efficiency 
of the algorithm to identify the decays of heavy hadrons, as at least two well- 
measured tracks are required to form a displaced vertex, and the presence of hits 
in multiple silicon layers in the inner tracking volume improves the chances of 
finding that the vertex is significantly displaced from the primary. The tails in 
1672 the impact parameter resolution — due to hard nuclear collisions with detector 
material, multiple scattering, and mis-assigned hits in the silicon detector and 
COT — determine the false tag rate. 
1675 The efficiency of the algorithm to identify heavy hadrons increases with the 
hadron momentum. The efficiency is not measured directly since not all hadron 
decay products are reconstructed; instead it is parameterized as a function of 
the transverse energy Et of the jet and shown in Figure [531 The efficiency 
rises with momentum because the tracks that result from the decay are better 
measured at higher energy, and the decay flight distance is also longer. At very 
high Et, the tagging efficiency drops, as tracks begin to share hits, with the 
jets becoming more collimated. Figure [23] also shows the 6-tag efficiency as a 
function of the pseudorapidity, \rj\, of the jet. Jets at higher pseudorapidity pass 
through more material and have fewer COT hits, reducing the tag efficiency and 
less raising the false tag rate as shown in Figure 1241 At very high pseudorapidity, 
1686 the loss of tracking efficiency in the COT also reduces the false tag rate. 

At high instantaneous luminosities, the average occupancy of the silicon de- 
tector and the inner layers of the COT rises. This leads to increased chance of 
assigning noise hits to the tracks as well as missed COT hits due to ambiguity 
in resolving many overlapping tracks. As a result, at higher instantaneous lumi- 
nosities, the 6-tag efficiency drops slightly and the fake tag rate rises. The 6-tag 
efficiency is shown in Figure [25] as a function of the number of reconstructed 
primary vertices per beam crossing, which is more directly related to the track- 
ing occupancy than the instantaneous luminosity. The 6-tagging performance is 

1695 remarkably robust at high luminosities due to the high granularity of the silicon 

1696 detector covering the low radii tracking region. 

1697 10.3. SVT Efficiency Study 

1698 Aging of and radiation damage to the silicon detector resulted in increased 

1699 noise and reduction of the level of the signal (see Section l§~4")) . These factors 

1700 degraded the performance and efficiency of the SVT (see Section 13. 1 . If) . A 

1701 study was performed during the Tevatron run to measure the SVT efficiency as 

1702 a function of total integrated luminosity and estimate any potential impact on 
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Figure 23: Efficiency of the displaced vertex b-tagger, as functions of jet Et and jet pseu- 
dorapidity, for two configurations of the b-tagger. The efficiency is obtained from tagging 
jets which have been matched to b quarks in Monte Carlo top quark decays, multiplied by 
data/MC scale factors. 




Figure 24: Probability of non-6-jets to be 6-tagged for the displaced vertex b-tagger, as func- 
tions of jet E^ and jet pscudorapidity, for two configurations of the 6-tagger. The probabilities 
have been measured from inclusive jet data. 



1703 physics analyses. Starting from the measured level of signal and noise in the 

1704 silicon detector for a reference data sample at 3 fb , simulated samples were 

1705 produced by applying extrapolations of the signal and noise up to 8 fb -1 . Each 

1706 of the samples was used as input to SVT simulation software to estimate the 

1707 efficiency of the trigger at the corresponding integrated luminosity. Figure [26] 

1708 shows the prediction of the SVT efficiency as a function of total integrated 

1709 luminosity for four (open circles) and all five (closed circles) layers of SVX-II. 

1710 A decrease of SVT efficiency of about 4% was predicted between 3-8 fb -1 , 

1711 while the impact of losing an SVX-II layer was about 13%. 



1712 11. Summary 

1713 The CDF silicon detector, consisting of the SVX-II, ISL, and LOO com- 
17W ponents, was designed to withstand only 2-3 fb -1 of integrated pp collision 

1715 luminosity and was expected to be replaced in 2004 by an upgrade. It ran suc- 

1716 cessfully for over 10 years, through 2011, and was exposed to about 12 fb -1 of 
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Figure 25: Efficiency of the displaced vertex fe-tagger, as a function of the number of recon- 
structed pp collision vertices per beam crossing, for two configurations of the 6-tagger. The 
efficiency is obtained from tagging jets which have been matched to b quarks in Monte Carlo 
top quark decays, multiplied by data/MC scale factors. 
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Figure 26: Prediction of the efficiency of SVT as a function of total integrated luminosity. 
Closed black circles show the efficiency when only four layers of SVX-II are used by the SVT 
algorithm; empty red circles show the efficiency including all five layers. 



1717 integrated luminosity. About 90% of its ladders took data with high efficiency 

1718 until the end of Tevatron Run II. This was an unprecedented feat compared to 

1719 any silicon detector in the same category prior to it. It was also the first silicon 

1720 detector system to be incorporated into a hardware trigger to identify tracks 

1721 from secondary vertices. It provided precise measurements of the trajectories of 

1722 charged particles which were important to identify and measure heavy-flavored 

1723 hadrons, which in turn were crucial to CDF's physics program, including top 

1724 quark, b hadron, and Higgs boson physics. 

1725 The detector consisted of about 722,000 readout channels, with approxi- 

1726 mately 500 independent ladders, which required voltages to bias the sensors as 

1727 well as run the data acquisition electronics mounted on the detector. Elabo- 

1728 rate data acquisition, trigger, cooling, and monitoring systems were required 

1729 to collect the data used for analysis. The detector itself was located inside the 

1730 drift chamber in a volume heavily congested with cabling, cooling pipes, and the 

1731 beam pipe, that rendered it largely inaccessible for repair. Due to its inherent 
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1732 complexity, the detector operations involved detailed procedures and required 

1733 specially trained experts to execute them. Keeping up with the loss of experts 

1734 and training new ones to replace them was one of the major challenges faced 

1735 by the silicon operations team. In addition, inaccessibility of the detector re- 

1736 quired that every detector access be planned elaborately and well in advance 

1737 to take advantage of the Tevatron shutdowns. Future experiments must take 

1738 extra care in designing the accessibility aspects of their subsystems to avoid 

1739 these situations. 

1740 The silicon detector system had its share of failure modes, most of which 

1741 were addressed during commissioning and the rest were mitigated during the 

1742 operation of the detector. The most serious failure modes, those that required 

1743 immediate response, such as power supply failures, cooling system failures, and 

1744 unsafe beam conditions, had dedicated hardware and software systems designed 

1745 to protect the silicon detector from damage. 

1746 Unanticipated failure modes and exposures to detector damage became ev- 

1747 ident as Run II of the Tevatron progressed. These had been addressed with 

1748 hardware modifications, monitoring, review, and improved operational proce- 

1749 dures to reduce the chance of damage to the detector. For example, the Lorentz 

1750 force on bond wires perpendicular to the magnetic field, connecting one side 

1751 of a ladder to another, on rare instances caused the wires to vibrate at their 

1752 mechanical resonant frequency and eventually break (Section 13.61) . A hardware 

1753 device was devised and installed to stop data acquisition when the trigger system 

1754 requested readouts at high frequencies and at regular intervals. Another issue 

1755 involved spontaneous energizing of the kicker magnets or separators sparking at 

1756 unexpected times, which steer the beams near the silicon detector and caused 

1757 damage to it due to the acute radiation dose ( Section I6.1.2[) . The addition of 

1758 a collimator near CDF and improvement to the high voltage conditioning of 

1759 the separators had minimized the effect of such incidents on the silicon detec- 

1760 tor. Careful monitoring of beam conditions, automatic ramp-down of the bias 

1761 voltage in case of bad beam conditions and beam abort requests from the dia- 

1762 mond sensors had also protected the silicon detector from damage due to beam 

1763 incidents. 

1764 Certain electronic components necessary for the functioning of the silicon 

1765 detector were located in the collision hall. Radiation caused temporary and 

1766 permanent failures of data acquisition electronics, particularly FPGAs (Sec- 

1767 tion l3.8.Ij) . The power supplies were also susceptible to radiation-induced fail- 

1768 ures (Section l4.3p . Many of the temporary failures could be addressed simply by 

1769 resetting and re-initializing the affected components, and these procedures were 

1770 highly automated so that the reset and recovery to the nominal data-taking 

1771 configuration resulted in a minimum of downtime. Permanent failures of col- 

1772 lision hall electronics typically required short accesses to replace the affected 

1773 components. A sufficiently large pool of spares was kept on hand to maintain 

1774 a high availability of the detector. Many of the components were repaired at 

1775 Fermilab, and others required sending equipment back to the manufacturer. 

1776 The cooling system for SVX-II and LOO worked remarkably well, while that 

1777 of ISL experienced higher failure rates (Section [5]). The initial epoxy blockages 
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1778 were cleared with laser light guided inside the small cooling tubes with fiber 

1779 optics. A later incident stemming from acidified ISL coolant caused leaks in the 

1780 piping cooling the portcards. These leaks were sealed with epoxy from inside the 

1781 pipes as the outside was inaccessible. The ISL coolant was returned to distilled 

1782 water, as in early running phase, and care was taken to monitor and respond 

1783 rapidly to changes in the cooling system. Nonetheless, as the system aged, leaks 

1784 in the piping became larger. These did not impact the operation as the coolant 

1785 ran below atmospheric pressure. 

1786 The unexpectedly high longevity of the silicon detector, which came as a 

1787 welcome surprise, is in part due to the slow aging of the sensors as radiation dose 

1788 was accumulated (Section [5]). The inner layers of the detector type- inverted as 

1789 expected. The depletion voltage, signal response and noise behaved as expected 

1790 also after type inversion. There was evidence that the electric field was not a 

1791 linear function of the position within the sensor, but instead strengthened near 

1792 the p+ and n+ implants. This created two depletion regions, one on either 

1793 side of an underdepleted bulk. Sensors that were not fully depleted at the end 

1794 of Run-II still provided usable data on both sides of the sensor, with slightly 

1795 reduced charge collecton on the p-side. 

1796 As a result of to its size and complexity, the CDF silicon detector required a 

1797 dedicated team of experts to operate and maintain it, ensuring the continuous 

1798 harvest of high quality data. Despite the challenges from a prolonged run and 

1799 the gradual reduction of spares and experts toward the end of Run II, the 
lsoo data taking was concluded successfully due to extensive efforts on procedure 
lsoi automation and diligent monitoring of every subsystem. Sufficient experience 

1802 with detector aging, operational failure modes and their mitigation gained along 

1803 the way ensured good performance during the final years of Tevatron running 

1804 at high luminosity. Many profound successes of the CDF physics program were 

1805 the direct result of the high quality data provided by the silicon detector and 

1806 the dedicated displaced vertex triggers based on it. 
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